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Dear Readers, i | 
his issue of BSD Magazine is dedicated to boot 
environments, including the article on ZFS Boot 

Environments by Kris Moore. Moreover, you can read 

articles about email gateway, service spawner, web 

programming, PKGNG and Apache THRIFT. 

We start with the Let’s Talk section, where Rob 
Somerville shares his thoughts on “opt in” legislation 
for access to adult material. % 

Next, in the Developer’s Corner, we take a 
look at the article about Apache THRIFT by 
Chirag Maheshwari. 

Then, in the What’s New section, we take a 
look at an article about ZFS Boot Environments by 
Kris Moore. nae 


Alexandro Silva describes how to build an email 
gateway with FreeBSD to prevent malware and 
undesirable messages. 

Next, Daniel Dettlaff talks a bit about The Service 
Spawner that does user-side installation, software 
configuration, and maintains your software automatically. 

This month's Admin section continues with the 
menu navigation system and the usage of Javascript. 

Finally, Joe Maloney talks about PKGNG: 
The future of packages on FreeBSD and PC-BSD. 


We hope you enjoy this issue and find many 
interesting articles inside! 


Kamil Sobieraj 


Editor of BSD Magazine 
& BSD Team 
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Let’s Talk 


OG Quis custodiet ipsos custodes (Who will 
guard the guards themselves)? 
By Rob Somerville 
With the UK government in alliance with Internet Service 
Providers determined to introduce “opt in” legislation for 
access to adult material, has the Net come of age or are 
we entering a new period of censorship? 


Developer’s Corner 


OSApache THRIFT: A much needed tutorial 
By Chirag Maheshwari 
This is the first article on Apache Thrift, as there is 
neither any official documentation nor any tutorial 
available for the fabulous tool Apache Thrift. This article 
tries to bridge that gap and introduce you to Apache Thrift 
and how, when and why you should use it. 
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What's New 


= OA closer look at the changes in 
PC-BSD/TrueOS 9.2 — Part 1 
— ZFS Boot Environments 
By Kris Moore 
With PC-BSD 9.2 just around the corner, this is a good time 
to take an advanced look at some of the new functionality 
coming with it. While there are a number of new features in the 
works, today we will look specifically at the implementation 
of ZFS Boot Environments. In coming issues, we will also 
look at PC-BSD’s home directory encryption via PEFS and 
its PKGNG support. Let us get started by first looking at ZFS 
Boot Environments and the beadm command. 


= <An email gateway with FreeBSD to 
prevent malware and undesirable 
messages 
By Alexandro Silva 
Controlling inbound and outbound mail messages is a big 
challenge for sysadmins. Malware can spread quickly, 
infecting dozens of mailboxes and relentless soammers 
send thousands of messages with unsolicited advertising 
and phishing scams. 


=S The Service Spawner 

By Daniel Dettlaff 
lf you have ever dreamed of software that does user-side 
installation, software configuration, and maintains your 
software automatically without the need for a vast amount 
of UNIX systems knowledge, you should probably take a 
look at the universal Service Spawner. 


Admin 


<3 44 FreeBSD Programming Primer — Part 7 
By Rob Somerville 

In the seventh part of our series on programming, we 

will continue with the menu navigation system and using 

Javascript. 


<= PKGNG: The future of packages on 
FreeBSD and PC-BSD 


By Joe Maloney 

This article will show how to install, upgrade and remove 
packages using pkgng. It will also discuss some of the 
improvements over pkg “tools and demonstrate how 
pkgng will benefit end users. In addition, upcoming 
functionality of pkgng will also be briefly discussed. Finally, 
it will show how to install Gnome 3 and Cinnamon on 
PC-BSD Rolling Release using Pkgdemon. 
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LET’S TALK 


Quis Custodiet Ipsos 


Custodes 


(Who will guard the guards themselves)? 


With the UK government in alliance with Internet Service Providers 
determined to introduce “opt in” legislation for access to adult 
material, has the Net come of age or are we entering a new period 


of censorship? 


typed, on the blogo-sphere arguing the pros and 
cons of filtering pornographic material via “Opt In” 
at the UK ISP's level. | will take a very British view on 
this, in so much that what two consenting adults do in the 
privacy of their own home — provided it is legal and no 
harm becomes of others — is of no concern to others. At 
the end of the day, as adults we have a conscience to 
guide us, and indeed the foundation of Western criminal 
law is based on mens rea — the act is not culpable unless 
the mind is guilty. So | will not add to the noise by debat- 
ing the rights or wrongs of a piece of forthcoming legisla- 
tion which seems based on a populist knee jerk reaction 
— however well intentioned it may be. | suspect history will 
prove it to be ineffectual, a political “get out of jail free” 
card that absolves those in positions of responsibility by 
allowing them to say with mealy mouth, “at least we tried”. 
The scenario goes much deeper than just protecting 
children. The definition of a civilised society is how well 
we look after the poor, the sick, and those that cannot 
look after themselves. No right thinking adult could argue 
with that premise. But what happens when the law of un- 
intended consequences takes hold? The proponents of 
this legislation would argue that it is not censorship — but 
they cannot argue against the fact that hooks are being 
put in place to potentially control those that choose to ac- 
cess adult material — whether sexual or otherwise. In oth- 
er words, the camel’s nose of censorship is now well in- 
side the tent or to use a very English phrase, this is just 
the thin edge of the wedge. 
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The potential for abuse is already evident from the adult 
filters on mobile networks in the UK. Gun, alternative news 
political, community and technology sites and those with 
“Inappropriate” content are all lumped together with the 
more carnal purveyors of adult entertainment. To add in- 
sult to injury, the filters are either on or off. You can either 
receive a bland diet of what our moral guardians believe 
is “safe” or the whole gamut, including the most distaste- 
ful the Internet has to offer. This is not a hypothetical list — 
these sites have already been blocked in the UK by adult 
filtering. The difficulty is that the appeal process is unmer- 
ciful, and it is doubtful that in law any web-master hav- 
ing his or her site incorrectly included in the list would be 
able to reclaim loss of income or reputation via the courts. 
And let’s be honest, the technology isn't there 
to truly determine the really nasty stuff. 
So the potential to monetise freedom 
has now arrived at a computer near 
you. Those behind the curtain yet 
again hold the power but escape 
accountability. 

Once the majority of people re- 
alise this, they will vote with their 
feet and choose to have the fil- 
ter removed. Then what? Your 
local friendly ISP (Or worse 
still some quasi-government 
agency) will have a _ sub- 
scribers list that marketeers, 
blackmailers, the media, law 
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enforcement agencies and other interested parties would 
give their right arm for. Human nature being what it is, this 
information will get out. Be it through commercial forces 
(Sell us your database, we'll give you a pound per pervert) 
or “public interest” (Political opponent X enjoys watching 
“Adult” movies) the vista for abuse and control is limitless. 
As for black hat hackers, you get the point. 

The problem, of course, is that the Internet needs to po- 
lice itself rather than abrogating responsibility to lawyers, 
politicians and others who wouldn't recognise an encrypt- 
ed tunnel or a VPN if it fell on them with the router still 
attached via steel cable rather than CAT 6. As a commu- 
nity, we need to expose those that damage the reputation 
of the Internet, but unfortunately the political will just isn’t 
there to deal with the real offenders. A case in point — | re- 
cently came across a live phishing email on a Friday eve- 
ning pointing to a major UK bank. The bank wasn't inter- 
ested, and if | had contacted the police, | would have been 
given short shrift. The ISP responsible wasn't that both- 
ered either. We live in a 24/7 culture, and we need a 24/7 
response to the threats that materialise and disappear just 
as quickly. Of course, vexatious complaints would be a 
problem, but the community has a way of identifying and 
isolating those that are troublemakers. Please don’t feed 
the trolls. While a global Internet police force would be a 
bad thing, we do need some sort of mechanism free of 
political and commercial bias that has teeth. And it needs 
to work both ways — If Mr. Dictator decides his populace 
cannot view content from another country we do not have 
a World Wide Web, we have a filtered LAN. No comfort- 
able deals made in smoke-filled rooms. 

We are sliding towards a crisis of medieval propor- 
tions where those with vested interests can control not 
only what the public can or cannot read and view but 
also when and how. DRM and the threadbare Intellec- 
tual Property rights argument are only the beginning of 

the pogrom against Internet and content freedom. 
Rather than dealing with the minority of 

bad apples (commercial, government 
and private), the whole barrel is 
tarred with the same brush. 
If all the energy that was 
dissipated in deal- 

ing with torrents 

and illegal file 

sharing was 

directed _to- 

wards corrup- 

tion that costs 

people their pen- 

sions, livelihood — 


a | 


and in developing countries, their lives — the world would 
be a better place. But hey, that would affect our bottom 
line. Go for the easy solution rather than dealing with the 
true perpetrators. 

In the age of technology, the Internet remains the last 
bastion of true democracy. What is done in the dark needs 
to be exposed, and that cuts both ways. The whole pano- 
ply of human character is available on the web, as it is 
in any gathering of people. It is too late to argue if the 
community has lost the technological or PR battle; the 
legislators are gearing up and the commercial and legal 
realities will mean any compromised organisations will 
crumple. Google has already been in front of the select 
committee for tax avoidance in the UK, and strategically 
where Google goes, a significant percentage will follow. 
To the community's chagrin, Google has capitulated free- 
dom while the East probes Western servers with impunity. 
True freedom and democracy are scary responsibilities, 
but as usual, a pragmatic political compromise leaves an 
unpleasant taste in the mouth. Idealism has a short shelf 
life where hard currency is involved. 

Tragically, the community is faced with a paradox that 
embraces all innovators, visionaries and growing societ- 
les — how do we police ourselves without delegating re- 
sponsibility? More challenging still, how can we prevent 
our idealism from being corrupted by those that would 
seek to capitalise on human weakness and vulnerability 
to their advantage? In the year 2013 with scandals affect- 
ing the financial markets, traditional institutions and in- 
deed nations, the Internet community is not alone in ask- 
ing these questions. 

Ultimately, the whole Internet is stronger than the sum of 
its parts. Those in the dark have been running scared, as 
has been proven when the kill switch has been activated 
on previous occasions. If we can maintain the moral high 
ground, police ourselves with integrity, hopefully we can 
shrug off the pseudo-ethical straitjacket those that do not 
understand technology seek to impose. 


ROB SOMERVILLE 

Rob Somerville has been passionate about technology since 
his early teens. A keen advocate of open systems since the mid 
eighties, he has worked in many corporate sectors including fi- 
nance, automotive, airlines, government and media in a vari- 
ety of roles from technical support, system administrator, de- 
veloper, systems integrator and IT manager. He has moved on 
from CP/M and nixie tubes but keeps a soldering iron handy 
just in case. 
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Apache THRIFT: A Much 


Needed Tutorial 


This is the first article on Apache Thrift, as there is neither 
any official documentation nor any tutorial available for the 
fabulous tool Apache Thrift. This article tries to bridge that 
gap and introduce you to Apache Thrift and how, when and 


why you should use it. 


What you will learn... 

« What is Apache Thrift 

« Why we should use Apache Thrift 
« How to use Apache Thrift 

¢« Comparison with similar tools 

¢ Howitcan be scaled 


be used. It also tries to address the challenge of 
scalability by discussing how Thrift can be used to 
meet enterprise expectations. 


Ty his article is about Apache Thrift and how it can 


What is Apache Thrift 
From the Thrift Website (http://thrift.apache.org/): 

Thrift is a software framework for scalable cross-lan- 
Quage services development. It combines a software 
stack with a code generation engine to build services that 
works efficiently and seamlessly between C++, Java, Py- 
thon, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Ja- 
vaScript, Node.js, Smalltalk, and OCaml. 

Thrift is an Interface Definition Language (IDL) which 
is used to define and create services between numerous 
languages as a Remote Procedure Call (RPC). Its light- 
weight framework and support for cross-language com- 
munication makes it more robust and efficient than other 
RPC frameworks like SOA (REST/SOAP) for most of the 
operations. Through a simple and straightforward IDL, it 
allows you to create services that are usable by numerous 
languages. Using code generation, Thrift creates a set of 
files that can be used to create clients and/or servers. In 
addition to interoperability, Thrift can be very efficient be- 
cause of a serialization mechanism which can save both 
space and time. 
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What you should know... 
¢ basic shell commands 
« basic knowledge of programming 


In other words, Apache Thrift lets you create a service to 
send/receive data between two or more softwares that are 
written in completely different languages/platforms. 


History 
Thrift was originally developed by the folks at Facebook. 
It's also one of the “core parts of their infrastructure”. Af- 
ter a while, they decided to make it Open Source and fi- 
nally contributed it to Apache Software Foundation (ASF) 
in April 2007 in order to increase usage and development. 
Thrift was then released under the Apache 2.0 license. 
The choice of programming language at Facebook is based 
on what language is best suited for the task at hand. While 
pragmatic, this flexibility resulted in difficulties when these ap- 
plications needed to call one another. After some analysis, 
Facebook engineers did not find anything currently existing 
which could meet their needs of interoperability, transport ef- 
ficiency, and simplicity amongst others. Out of this need, they 


Search Service 
(implemented in C++ 


PHP based Thnft PHP Lib 


Figure 1. Facebook Search Service Architecture 
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developed efficient protocols and a service infrastructure that 
became Thrift. Facebook now uses Thrift for their back-end 
services, the reason for which it was initially designed. 


¢ Facebook Thrift is one of the core components of 
Facebook infrastructure. Its search services are im- 
plemented in C++, but its Web Application is based 
on PhP. Thus, to bridge the gap between them, Face- 
book uses Thrift (Figure 1). 

¢ Evernote Thrift is extensively used in a variety of the 
Evernote public API. 

¢ Scribe is also built on top of Thrift. 

¢ HBase leverages Thrift for cross-language API. 

¢ The whole list can be found here: htto://wiki.apache. 
org/thrift/PoweredBy 


The stable release of Apache Thrift can be downloaded 
from: http://thrift.apache.org/download/. 


Listing 1. /nstalling Thrift on Debian/Ubuntu 


##Install the pre-requisites 

Sudo epr-der incscall Mubspoost-dey dibboos:—resr— 
dev libboost-program-options-dev 
libevent-dev automake libtool flex 
louLSeial jolecj-Ceimne; Char lisse loc ey 

Sudo apbe-geu imstall phpo-dev johps-cli 

Can ue emp, 

fetch “http://www.trieuvan.com/apache/thrift/0.9.0/ 
iBiduetb ce 0)s, Sally ese erat 


You can talk the talk. 
Can you walk the walk? 


tar xvzf thrift- [ IT’S IN YOUR DNA ] 
Col 1eleasne= 
./pootstrap.sh LEARN: 

. Advancing Computer Science 
_ Connileyb ible Artificial Life Programming 
sudo make Digital Media 
sudo make install Digital Video 


Enterprise Software Development 


Listing 2. /nstalling Thrift on FreeBSD Game Art and Animation 


Game Design 
##Install the pre-requisites and then, Game Programming 
##0n freeBSD, Thrift can be installed from FreeBSD Human-Computer Interaction 
; Network Engineering 
ea Network Security 
Open Source Technologies 
@cly/ Uct/ ports) devel/thriit Robotics and Embedded Systems 
Ib serious Game and Simulation 
P/ DOCS iaclor sill ; | 
. Strategic Technology Development 
- /configure Technology Forensics 
make Technology Product Design 


Technology Studies 
Virtual Modeling and Design 
Web and Social Media Technologies 


make install 
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Install on Ubuntu 
Commands are given in Listing 1. 


Install on FreeBSD 
Commands are given in Listing 2. 


Install on Windows 


¢ First install some pre-requisites 


¢ MinGW 

¢ GNU Build Tools 
e gt+ 4.0+ 

¢ bison 2.3-1 


¢ boost 1.33.1-4 
¢ boost-devel 1.33.1-4 


¢ flex 2.5.33-1 
¢ pkgconfig 
¢ libtool 


¢ Download thrift.exe 

¢ Copy it to C:\Thrift\thrift.exe 

¢ Now, add “C:\Thrift\’ to your PATH environment vari- 
able 

¢ Now the thrift compiler can directly be used from 
CMD prompt. 


Install Thrift Editor for Eclipse 


¢ Open Eclipse 
¢ Eclipse —> help —> Install new Software 


Client Server 


a ) 
»Generated¢ 


a Code ‘ 


Figure 2. Apache Thrift client-server architecture 
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¢ Add the URL: http:/thrift4eclipse.sourceforge.net/up- 
datesite/ 

¢ Check the only available package 

¢ Install the package 


Architecture 
Thrift includes a complete stack for creating clients and 
servers. Figure 2 depicts the Thrift Stack. 

The top portion of the stack is generated code from 
your Thrift definition file. Thrift services result in gener- 
ated client and processor code. These are represented 
by the brown boxes in the figure. The data structures that 
are sent (other than built-in types) also result in generat- 
ed code. These result in the red boxes. The protocol and 
transport are part of the Thrift runtime library. Therefore 
with Thrift, you can define a service and have the freedom 
to change the protocol and transport without regenerating 
your code. Thrift also includes a server infrastructure to tie 
the protocols and transports together. There are blocking, 
non-blocking, single and multithreaded servers available. 
The “Underlying I/O” portion of the stack differs based on 
the language in question. For Java and Python network 
I/O, the built-in libraries are leveraged by the Thrift library, 
while the C++ implementation uses its own custom imple- 
mentation. 

Thrift allows you to choose independently between your 
protocol, transport and server. With Thrift being originally 
developed in C++, Thrift has the greatest variation among 
these in the C++ implementation. 


Transport Layer 

The transport layer provides simple abstraction for read- 
ing/writing to/from the network. The transport layer ba- 
sically describes “how” data is transmitted. This layer 
decouples the underlying transport from the rest of the 
system, exposing only the following interface: 


* open 
¢ close 
°* read 

°¢ write 
¢ flush 


In addition to the above interface, Thrift also uses Ser- 
verlransport interface on the server side to accept or 
create transport objects. The interface includes: 


* open 
e listen 
° accept 
¢ close 


08/2013 


Apache THRIFT: A Much Needed Tutorial 


There are a number of transports supported by Thrift: 


¢ TSocket: Uses blocking socket I/O for transport. 

¢ TFramedtTransport: Sends data in frames, where 
each frame is preceded by a length. This transport is 
required by a non-blocking server. 

¢ TFileTransport: This transport writes to a file. This 
transport is not included with the Java implementation. 

¢ TMemoryTransport: Uses memory for I/O. The Ja- 
va implementation uses a simple ByteArrayOutput- 
Stream internally. 

¢ TZlibTransport: Performs compression using zlib. It 
should be used in conjunction with another transport. 
Not available in the Java implementation. 


Protocol Layer 

The protocol abstraction defines a mechanism to map in- 
memory data structures to a wire-format. It specifies how 
datatypes use the underlying Transport to encode/decode 
themselves. Separates Data Structure from Transport repre- 
sentation. Thus the protocol implementation governs the en- 
coding scheme and is responsible for (de)serialization. Thrift 
protocols are stream oriented by nature thus there is no need 
for any explicit framing. In other words, Protocols describe 
“WHAT” is actually transmitted. Thrift supports both text and 
binary protocols. The binary protocols almost always outper- 
forms text protocols, but sometimes text protocols may prove 
to be useful in cases of debugging. The Protocols available 
for the majority of the Thrift-supported languages are: 


TBinaryProtocol 

A straightforward binary format encoding numeric values 
as binary, rather than converting to text. They are not op- 
timized for space efficiency. 


TCompactProtocol 

Very efficient and dense encoding of data. This protocol 
writes numeric tags for each piece of data. The recipient is 
expected to properly match these tags with the data (If the 
data is not present, no tag/data pair is present). For Inte- 
gers, this protocol performs compression using Variable- 
Length Quantity (VLQ) encoding from the MIDI file format. 


TDenseProtocol 
It's similar to TCompactProtocol but strips off the meta 
information from what is transmitted and adds it back at 
the receiver side. It is still experimental and not yet imple- 
mented in Java. 


TJSONProtocol 
Uses JSON for data encoding. 
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TSimpleJSONProtocol 
A write-only protocol using JSON. Suitable for parsing by 
scripting languages. 


TDebugProtocol 

Sends data in the form of human-readable text format. 
It can be well used in debugging applications involving 
Thrift. 


Processor Layer 

A processor encapsulates the ability to read data from in- 
put streams and write to output streams. The input and 
output streams are represented by protocol objects. The 
processor interface is extremely simple. Service-spe- 
cific processor implementations are generated by the 
Thrift compiler. 

Thus, the generated code makes the Process Lay- 
er of the architecture stack. The processor essentially 
reads data from the wire (using the input protocol), del- 
egates processing to the handler (implemented by the 
user), and writes the response over the wire (using the 
output protocol). 


Server Layer 

A server pulls together all of the various functionalities de- 
scribed above to complete the Thrift server. First, it cre- 
ates a transport and specifies input/output protocols for 


Listing 3. fle: hello.thrift 


namespace java helloworld 
service HelloService { 


string sayHello() 


Figure 3. Thrift Application Development flow-chart 
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Listing 4. Generating Processors in Thrift 


fey 


##Generating the processor for Server Side 
Enritie=--gen java hello zrnrire 


##Generating the processor for Client Side 


Uhritt —=-gen php hello, thrite 


Listing 5. file: HelloServicelmpl.java implementing the hello world 


service 


package helloworld; 


import org.apache.thrift.TException; 


public class HelloServiceImpl implements HelloService.Iface { 


public String sayHello() throws TException { 


return “Hello World!!”; 


Listing 6. file: HelloServer.java which is code for Java server. 


package helloworld; 


import org.apache.thrift.server.TServer; 
import org.apache.thrift.server.TThreadPoolServer; 


import org.apache.thrift.transport.TServerSocket; 


Import Org spache Er iih bs EmamMcoOm Ew mans pomiel eee Len, 


public class HelloServer implements Runnable { 


Publie State yoid maan(string i) args) 4 


new Thread(new HelloServer()).run(); 


Listing 7. file: client.php, Client side code. 

<?php 
// defining the port and server to listen 
define (“PORT”, ‘9090’); 


define (“SERVER”, ‘localhost’); 


//Global variable where the php library files are stored 


~GLOBALS [| “THRIFT ROOT’ | = ‘thrift’; 


//including the library files 


require once $GLOBALS[ ‘T RieD ROOTS.) /Tarrtc. 
require once $GLOBALS[ ‘THRIFT ROOT’].’/protocol/ 


n 


[BinaryProtocol.pnp ¢ 


ae | 


require once $GLOBALS[ ‘THRIFT ROOT’].’/transport/ 


LSOCKet. phe 7 


require once $GLOBALS[ ‘THRIFT ROOT’].’/transport/ 


TBufferedTransport.php’ ; 


//loading the auto-generated package 


require once $GLOBALS[ ‘THRIFT ROOT’].’/packages/ 


hello/HelloService.php’ ; 


php’; 


ee 
private static final int PORT = 9090; <2 plap 
Ly | 
public word. cum() { j/fereate a thrift connection 
try < Ssocket = new TSocket (SERVER, PORT) ; 
TServerSocket serverTransport = new Stransport = new TBufferedTransport (Ssocket) ; 
TServerSocket (PORT) ; Sprotocol = new TBinaryProtocol (Stransport) ; 
HelloService.Processor processor = new 
HelloService. Processor (new //create a new hello service client 
HelloServiceImpl ()); Sclient = new HelloServiceClient (Sprotocol) ; 
TServer server = new TThreadPoolServer (new 
TThreadPoolServer.Args (serverTransport) . //open the connection 
processor (processor) ) ; pL Gans pOlE Omen jr 
System.out.println(“Starting server on port: “+PORT); 
server.serve(); Sresult = S$Sclient->sayHello(); 
echo “Result: “.Sresult; 
} catch(TTransportException e) { 
System.out.printin(“Message: “+e.getMessage()); SiC elec se LOSe |.) 4 
SVSEeMmeourpreintind( Strackirace:  “).; | Gatch (IEXceptiom sb) | 
2. printstackTrace(): echo “Thrift: Exception: “~~ Stx--getMescage (J. \r im; 
} } 
} a 
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the transport. Then, it creates a processor based on the 
I/O protocols. It finally waits for incoming connections. 
When a connection is made, it hands them off to the pro- 
cessor to handle the processing of the request. 

Thrift provides a number of servers: 


TSimpleServer 
A single-threaded server using standard blocking |/O 
socket. Mainly used for testing purposes. 


TThreadPoolServer 

A multi-threaded server with N worker threads using 
standard blocking I/O. It generally creates N=5 minimum 
threads in the pool if not specified otherwise. 


TNonBlockingServer 

A multi-threaded server using Non-Blocking IO (Java im- 
plementation uses NIO channels). TFramedTransport 
must be used with this server. 


THttpServer 
HTTP server (for JS clients) optionally with REST like URLs. 


TForkingServer 
Forks a process for each request to server. 


TProcessPoolServer 
Available in Python. Pre-forks workers to avoid Global In- 
terpreter Lock. 


Listing 8. Non-Blocking server in Java 


//Works with Asynchronous client too 


public class NonblockingServer { 


PLivake staple final Vom: PORT = 7/911; 
Private void start () { 
try { 
TNonblockingServerTransport serverTransport 
= new TNonblockingServerSocket (PORT) ; 
HelloService.Processor processor = 
new HelloService. Processor (new 


HelloServicelImpl ()); 


TServer server = new TNonblockingServer (new 
TNonblockingServer. 
Args (serverTransport). 
processor (processor)); 
System.out.println(“Starting server on port 
Was oe OSE neler NOent evo) ey ws ees) he 
server.serve(); 
} catch (TTransportException e) { 


e.printStackTrace(); 


Public Stable vord Maim(seriung |) args). 
NonblockingServer srv = new NonblockingServer () ; 


Ssry.start ()> 


Listing 9. Non-Blocking Client in Java 


//Note the usage of TFramedTransport for Non-Blocking 
Server which would frame the data 


/ Se Oe elas wads 


public class Nonblockingélvent | 
private void invoke() { 
Pires Oil wt aceite pod t - 
try { 
transport = new TFramedTransport (new 
Teochew lecalhosie . 7 oll) jr: 


TProtocol protocol = new TBinaryProtocol (transport) ; 


HelloService.Client client = new 
HelloService.Client (protocol) ; 


transport.open(); 


System.out.printin(client.sayHello()); 


teansporrsclose().; 

} catch (TTransportException e) { 
e.printStackTrace(); 

} catch (TException e) { 


e.printStackTrace(); 


DUbITe static void maim(Strineg ||) args) 
NonblockingClient c = new NonblockingClient(); 


C.invoke () > 
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Listing 10. Asynchronous Client written in Java 


//Uses the Non-Blocking Server given in Listing 8. 
//Note that for each operation of the service, a callback will be defined. Also, a new client will 


//nave to be used with every different operation else an exception will be thrown 


pubiite class AsyncClvenk { 


private void invoke() { 
try { 
HelloService.AsyncClient client = new HelloService. 
AsyncClient (new TBinaryProtocol.Factory(), new TAsyncClientManager(), 


new INonblockingSocket (“localhost”, 7911)); 


client.sayHello(new sayHelloMethodCallback()); 
} catch (TTransportException e) { 
e.printStackTrace(); 
} catch (TException e) { 
e.printStackTrace(); 
} catch (IOException e) { 


e.printStackTrace(); 


Publ C statie Vo1d Maim(SEring [| args) 4 
AsyncClient c = new AsyncClient(); 


c.invoke(); 


class sayHelloMethodCallback 
implements AsyncMethodCallback<HelloService.AsyncClient.sayHello call> { 


Pubic old onCompllere (Giellocer ice ae me Clrenm ca el loneallesayHelloucal ls) at 
try { 
Sicieme iestlbe = Senge Well CSsekesulbe()). 
System.out.println(“Result from server: “ + result); 
} catch (TException e) { 


e.printStackTrace(); 


public void onError(Exception e) { 
Shi Sicily elbhe Sjeveakiane lio Naieisie.e G0) 


e.printStackTrace(); 
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Thrift allows only one service per server. This is certainly 
a limitation and will be addressed later in this article un- 
der multiplexing. 


How to use? 
Figure 3 describes how the flow of a Thrift Application is 
made and Figure 4 shows the anatomy of a Thrift service. 

First we'll make a small “Hello World” application 
which aims to make RPC’s between Java (Server) and 
PHP (Client). 

We'll start by making a Thrift IDL file named “hello.thrift” 
as shown in Listing 3. The full tutorial for making a Thrift 
IDL file can be found here: http:/diwakergupta.github.1o/ 
thrift-missing-guide/thrift. pdf. 

Then we'll generate the files for processors layer using 
the Thrift compiler for both Client and Server using the 
commands given in Listing 4. 

Make a new project in Eclipse with type, “Dynam- 
ic Web Project”. Put the “thrift file’ in the <project- 
name>/Java/Resources/src/ directory. Copy the lib files 
(libthrift—<version>.jar, build/lib/*) tO <project- 
name>/WebContent/WEB-INF/1lib/ folder. Now, we have 
to implement the services mentioned in the thrift file. 
Write a new class named “HelloServicelmpl” imple- 
menting the “HelloService.lface” interface under the 
package “helloworld”. The code for the same is given 
in Listing 5. 

Now, we have to write the server which actually caters 
to the client's requests in Java. We will call this “Hel- 
loServer” implementing the “Runnable” interface. This 
class will also come under the package “helloworld”. The 
code for the same is given in Listing 6. As you can see, 
here we are. 

Now finally, we have to make the Client, which is writ- 
ten in PHP. We already generated the client side pro- 
cessor from the Thrift IDL. So first, we include all the 
Thrift run-time libraries by creating a new directo- 
ry named “thrift” and copy all the php library files avail- 
able in the directory /path/to/thrift-version-folder/ 
lib/php/src/ to the newly created directory. Also, cre- 
ate a new sub-directory named “packages” in “thrift” di- 
rectory and copy the auto-generated PHP package 
here. Create a new file <client-file>.php adjacent to the 
“thrift” directory. The contents of the PHP file are given 
In Listing 7. 

We now know how to make a basic Server-Client Appli- 
cation in Thrift which uses different language/platform for 
both the server and client. We will now try and explore the 
different available server and client combinations avail- 
able to us for improving performance. The previous exam- 
ple we gave was of Blocking Server and Client. Now we'll 
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make another server and client which is Non-Blocking in 
nature. The codes for Non-Blocking server and client are 
given in Listing 8 and Listing 9 respectively. 

Last but not least, we will try and implement an Asyn- 
chronous Server and Client. We can write asynchro- 
nous clients to call a Thrift service. A callback needs to 
be registered which will get invoked at successful com- 
pletion of the request. Blocking mode server didn’t work 
(method invocation returned with an empty response) 
with the asynchronous client (Maybe it’s because we 
are using TNonblockingSocket at the client side. See 
construction of HelloService.AsyncClient in Listing 10. 
So this may be the proper behaviour). Non-blocking 
mode server given in Listing 8 seems to work without 
an issue. So you can use the non blocking server from 
earlier with the client shown in Listing 10 to implement 
Asynchronous mode. 

So this is how we can use different combinations of 
transports, protocols, and servers to suit our needs with- 
out changing much of the code. 

It is now worth summarizing the pros and cons of the 
Thrift approach: 


Benefits 


¢ Cross-Language serialization with lower overhead 
than alternatives such as SOAP due to use of bina- 
ry format. 

¢ A lean and clean library. Neither any framework to 
code nor any XML configuration files. 

¢ The language bindings are very natural. For exam- 
ple, Java uSeS ArrayList<int> and similarly C++ uses 
Stdisvector<int>, 

¢ The application level wire format and the serialization 
level wire format are clearly separated. They can be 
modified independently. 

¢ Soft versioning of the protocol. Thrift does not require 
a centralized and explicit mechanism like major-ver- 
sion/minor-version. Loosely coupled teams can freely 
evolve RPC calls. 

¢ No build dependencies or non-standard software. No 
mix of incompatible software licenses. 

¢ Changing the current interface is very easy. The 
old Client or Server remain compatible with the new 
Server or Client respectively, silently ignoring any 
deprecated or newly added fields if necessary. 

¢ Thrift supports a wide variety of languages and envi- 
ronments. 

¢ Service inheritance, subservices implement all 
functions of their base services and can have addi- 
tional functions. 
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Limitations 


¢ Only one service per server. This can be addressed 
using multiplexing discussed in a short while but still 
adds to the complexity. 

¢ There can be no cyclic structs. Structs can only con- 
tain structs that have been declared before it. A struct 
also cannot contain itself. 

¢ Important OOP concepts like inheritance and poly- 
morphism are not supported. 

¢ Lacks full documentation. 

¢ Null cannot be returned by a server. Instead a wrap- 
per struct or value is expected. 

¢ No out-of-the-box authentication service available be- 
tween server and client. 

¢ No Bi-Directional messaging is available. 


Multiplexing of services 

Despite being a powerful and efficient cross-language com- 
munication tool, Thrift’s services are challenged by high ad- 
ministrative and maintenance overheads. The fact remains 
that every Thrift server is only capable of exposing only one 
service at a time. In order to host multiple functions, Thrift 
provides organizations with the following options: 


Context-Service 
Map 


Lookup 
Multiplexer 


Firewall 


Registry 
Client 


Client 


¢ Write a monolithic, unwieldy implementation and host 
it as a single service. 
¢ Host multiple small services across a series of ports. 


lf we follow the first option, writing monolithic services 
elevates the cost of development and maintenance as 
time passes since the complexity increases with the ad- 
dition of any new service. Even the second option can 
prove deadly in the long run as the number of ports con- 
sumed to host the multiple services keeps on increasing. 
Ports are a limited resource and need to be judicious- 
ly used. Clients will have to maintain too many connec- 
tions for each service they want to use. Also, aS many 
ports are opened, security needs to be properly scruti- 
nized, which introduces more overhead. 

Thus, we introduce multiplexing by extending the Thrift 
framework to create and host multiple services on each 
server. The baseline approach is to assign a symbolic 
name to each service which is referred to as “service con- 
text’. This will help us host multiple services on each serv- 
er where each service can be recognized by its respec- 
tive service context. The basic architecture can be seen 
in Figure 4. 

Let me explain the different components involved: 


Lookup 
Service 


Service 
Service-1 , 
Implementation 


Service-2 


Protocol 


MultiplexProtocol 


Transport 


Figure 4. Thrift Multiplexing architecture with registry lookup 
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Multiplexer 

This acts as the server side request breaker and identify- 
ing the service that the client has requested for, based on 
the service context provided by the client. This component 
maintains a mapping between the service context and the 
available services in that context. 


MultiplexProtocol 

lt is a wrapper class around the underlying protocol that is 
capable of embedding service context to the message on 
the client side and fetching it on the server side. 


RegistryLookup 

In order to reduce the overhead associated with manag- 
ing the service context manually, we have created a regis- 
try system that is responsible for managing meta informa- 
tion about the service context and their services. 


Servicelnformation 

This class captures/represents the information regarding 
services on a particular server. This object should be ca- 
pable of transmitting across the network and hence be 
used by the client to get service information. Service in- 
formation consists of service context, service name and 
description. 


MultiplexServer 

It is a new abstract server (wrapper to the old Server) 
which is capable of hosting any server implementation on 
any transport and protocol using TMultiplexLookup. It ba- 
sically provides an additional degree of freedom when it 
comes to hosting a new service on different transport and 
protocols with no additional coding effort. 


Table 1. Protocol efficiency study. 


Technology 


Message Size 
(The smaller TCompact 
the better) Protocol 


Thrift - TBinaryProtocol 445 71.15% 


% larger than 


Remote Method 880 238.46% 
Invocation 
REST — XML 809 211.15% 


Comparison With Similar Technologies 
As there was no data available on the internet about how 
different protocols perform against each other, | tried to 
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compare some of the available technologies for RPC 
between different languages, namely, Apache Thrift 
(TCompactProtocol, TBinaryProtocol), Google Proto- 
col Buffers, Java Remote Method Invocation, REST us- 
ing JSON and XML. These tests were conducted on a 
simple machine available at any home. To compare the 
protocols, | send out the same amount of data from the 
different technologies. | used WireShark to capture the 
sizes that were transferred between Client and Server. 
You can also perform the same exercise to see the per- 
formances for yourself. The results are tabulated and 
shown in Table 1. 

Thrift has a clear advantage in the size of its payload par- 
ticularly compared to RMI and XML-based REST. Protocol 
Buffers from Google are effectively the same given that the 
Protocol Buffers number excludes messaging overhead. 

To compare the runtime performances of the different 
technologies, | used the same computer to run both client 
and server whose specifications are as follows: 


¢ Operating System: Ubuntu Linux 12.04 

¢ System CPU: Intel Core i5-430M processor 

¢ Memory: 4GB 

¢ Java Version: Java(TM) SE Runtime Environment 
(build 1.7.0 _04-b20) 


| made around 100,000 queries from the Client monitor- 
ing the CPU usage and the total time taken. The results 
are tabulated and shown in Table 2. 


Table 2. Runtime Performance comparison 


CPU CPU 


Technology 


Average 
usage Usage Wall Time 
(SERVER) | (CLIENT) | (mmiss) 


Thrift - TBinaryProtocol 20% 33% 01:35 


Remote Method Invocation 46% 16% 02:18 


REST — XML 81% 12% 06:38 


The tests yielded some interesting observations. In terms 
of wall time, Thrift clearly outperformed REST and RMI. 
In fact, TCompactProtocol took less than 20% of the time 
it took REST-XML to transmit the same data. The clear 
dominance of the binary protocols should not be too sur- 
prising, as binary data transmission is well-known to have 
higher performance than text-based protocols. RMI in fact 
significantly out-performed JSON-based REST in wall 
time, despite its significantly larger payload size. 
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The CPU percentages yielded some interesting num- 
bers. While the Thrift and Protocol Buffers servers had 
the highest server CPU percentages, the REST clients 
had the highest CPU percentages of the clients. For 
whatever reason, Thrift and REST disproportionately 
place their CPU loads on their clients and servers. Pro- 
tocol Buffers balanced its load most evenly between cli- 
ent and server, but then again this was a simple quick 
hand-rolled server that | wrote for this article. While | 
did not have time to analyze the cause of the CPU load, 
the Thrift and Protocol Buffers examples needed to do 
manual conversion of objects between what is transmit- 
ted and what is used. The RMI and REST implementa- 
tions required no such object conversion. This extra bit 
of work may account for the additional CPU utilization 
on the Thrift and Protocol Buffers servers. 

Given the poor performance of REST, there may cer- 
tainly be higher performing servlet containers than Jetty 
that could be used as part of this test. Jetty was merely 
chosen because of its relative ease in implementation and 
ease in bundling the sample code used in this article for 
download. Doing some quick searches, | found one per- 
formance comparison that showed Apache Tomcat to be 
faster than Jetty and another that showed them at parity. 
Neither study showed anywhere near a performance dif- 
ference to make up for the wall time performance of the 
binary protocols. 

All of these technologies are roughly equivalent in the 
amount of coding complexity required to make them 
work. This excludes Protocol Buffers of course as it con- 
tains no services infrastructure. It should also be not- 
ed that Thrift generates all the code you need for a cli- 
ent or server for each language it supports. Java was 
the server of choice in this article, but other languag- 
es could be used if they are better suited — one of the 
main reasons Thrift was developed in the first place. 
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That being said, | found many of the implementations 
incomplete. As mentioned previously, the Python imple- 
mentation, for instance, only had the TBinaryProtocol 
implemented. 


Conclusion 

This article introduced you to the tool Apache Thrift and 
the different ways you can use it and shows how Apache 
Thrift can fulfill any needs with the flexibility it provides 
in choosing the different layers of the architecture sepa- 
rately. It outmatches any other similar technology avail- 
able, even Google protocol buffers, by providing more lan- 
guage support and flexibility. A sneak peek was given on 
how multiplexity can be used to tackle one of the limita- 
tions Apache Thrift has; a full implementation of it can be 
a whole article in itself and may be discussed in another 
article. 
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WHAT’S NEW 


A Closer Look at the 
Changes in PC-BSD/ 


TrueQOS 9.2. Part 1 


ZFS Boot Environments 


ith PC-BSD 9.2 just around the corner, this is a 
VV good time to take an advanced look at some of 

the new functionality coming with it. While there 
are a number of new features in the works, today we will 
look specifically at the implementation of ZFS Boot Envi- 
ronments. In coming issues, we will also look at PC-BSD’s 
home directory encryption via PEFS and its PKGNG sup- 
port. Let us get started by first looking at ZFS Boot Envi- 
ronments and the beadm Command. 


ZFS Boot Environments 

While ZFS boot environments have been around in Solar- 
is for a while, they are still relatively new to the FreeBSD 
ecosystem and many users may be unfamiliar with them. 
So what is a Boot environment (BE)? Simply put, it is a 
fancy way to leverage the power of ZFS snapshots to cre- 
ate instant system backups, which can be easily booted 
from in case of disaster. This is accomplished on PC-BSD 
and TrueOS by using a specific ZFS dataset layout, the 
beadm Utility, and the GRUB boot-loader. In order to make 
Boot Environments contain the proper data for backup 
/ restore, we have to start with a slightly different layout 
than a traditional ZFS layout (Figure 1). 


tank® 5.49G 28.7G 144K legacy 
tank0/ROOT 5.48G 28.7/G 144K legacy 
tank@/ROOT/default 5.486 28.76 5.486 /mnt 


Figure1. ZFS Layout of the “default” boot environment 
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The first change is that we are creating a special <tank>/ 
ROOT/default dataset. This dataset is the primary source 
for taking and cloning ZFS snapshots, which means any- 
thing outside of this dataset will not be included in a Boot 
Environment. It will be mounted as ‘/ on your system. 
However, for a Boot Environment to work, we will want to 
Include /usr, /usr/local, and such within our snapshot, 
while still allowing child datasets such as /usr/home and / 
usr/jails to be created. To accomplish this, we create a / 
usr ZFS dataset and set the canmount=of¢f flag. The same 
is done for the /var dataset, allowing us to include it in the 
snapshots, while creating /var/log, /var/tmp, and /var/ 
audit to persist between Boot Environments (Figure 2). 

These layout choices ensure that when we take a snap- 
shot of the system, we end up with all the system files and 
packages necessary to boot back up to a working desktop or 
server environment. Including /usr/1oca1 makes Boot Envi- 
2.66M 28./G6 152K 


tankO/usr/home fusr/hone 


tankO/usr/home/kris 2.51M 28.76 2.51M /usr/home/kris 
tank@/usr/ jails 144K «28.76 144K o/usr/jails 
tank@/usr/obj 144K 28.7G 144K $ /usr/obj 
tankO/usr/pbi 260K 28.7G 260K /usr/pbi 
tankO/usr/ports 296K 28.7G 152K /usr/ports 
tankO/usr/ports/distfiles 144K 28.7G 144K /usr/ports/distfiles 
tank@/usr/sre 144K 428.76 144K /usr/sre 
tank6/var B24K 28.76 144K Ss /mnt/var 
tank6/var/audit 160K 28.76 160K /var/audit 
tankO/var/log 368K 28.7G 368K /var/log 
tankO/var/tmp 152k 28.76 152K /var/tmp 


Figure 2. Additional ZFS file-systems excluded from a Boot 
Environment 
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ronments a good choice for backups before performing pack- 
age updates, because /usr/local updates are often just as 
critical to a user’s system as the kernel and world are. With 
the default layout ready for Boot Environments, we can now 
take a look at the beadm command to manage your backups. 
The beadm command, available in ports under sysutils/beadmn, 
provides an easy-to-use framework to create, destroy, and 
otherwise manage your various Boot Environments. The sub- 
commands we are most interested in will be /ist, create, and 
destroy. On a freshly installed system, you will start with a sin- 
gle Boot Environment named “default” (Figure 3). 

The “NR’ flags indicate that this BE is currently active 
Now, and will still be active on next Reboot. Creating a 
new BE will take a snapshot of the current BE at the cur- 
rent moment, so it would be wise to remember to do this 
before starting anything potentially dangerous. To add 
a new Boot Environment, simply run the beadm create 
<nickname> Command: Figure 4. 

As you can see, we have created a new BE nicknamed 
“newbe’”. A quick re-run of the list command will show the 
newly created environment, along with the time and cur- 
rent size cost (Figure 5). 

During the creation of the new BE, you probably saw the 
notice about generating the grub.cfg file. This brings us to 
the point of putting the “Boot” into Boot Environment. Sim- 
ilar to Solaris, PC-BSD and TrueOS 9.2 include the GRUB 
boot loader out of box and have it integrated into the beadm 
command in order to provide boot-time functionality. If we 
now reboot our system, we will be presented with a new 
Boot Environment menu: Figure 6. 

As you Can see, OUP newbe Boot Environment has been 
added to the menu, along with the creation date. By de- 
fault, a timer exists which will boot the first environment af- 
ter 5 seconds. However, if you press any key and interrupt 
this timer, you can arrow up and down to select the envi- 
ronment you wish to boot. Once you have selected the 
BE you wish to boot, GRUB will load the kernel + modules 


[root@pcbsd-8613] ~# beadm List 
BE Active Mountpoint Space Created 
default NR / 5.5G 2013-07-19 16:27 


Figure 3. Performing the initial list of Boot Environments 


[root@pcbsd-8613] ~# beadm create newbe 
Generating grub.cfg ... 

Found theme: /boot/grub/themes/pcbsd/theme.txt 
done 

Created successfully 


Figure 4. Creating a new Boot Environment 


[rootepebsd-8613] ~# beadm List 


BE Active Mountpoint Space Created 
default NR 5.5G 2013-07-19 16:27 
newbe - - 672.0K 2013-07-22 16:31 


Figure 5. Getting a listing of available Boot-Environments 
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The BSD Certification Group Inc. 
(BSDCG) is a non-profit organization 
committed to creating and 
maintaining a global certification 
standard for system administration 
on BSD based operating systems. 


@ WHAT CERTIFICATIONS ARE AVAILABLE? 


BSDA: Entry-level certification suited for candidates 
with a general Unix background and at least six months of 
experience with BSD systems. 


BSDP: Advanced certification for senior system administrators 
with at least three years of experience on BSD systems. 
Successful BSDP candidates are able to demonstrate 

strong to expert skills in BSD Unix system administration. 


@ WHERE CAN I GET CERTIFIED? 


We're pleased to announce that after 7 months of 
negotiations and the work required to make the exam 
available in a computer based format, that the BSDA 
exam is now available at several hundred testing centers 
around the world. Paper based BSDA exams cost $75 USD. 
Computer based BSDA exams cost $150 USD. The price of 
the BSDP exams are yet to be determined. 


Payments are made through our registration website: 
https://register.bsdcertification.org//register/payment 


@)_ WHERE CAN I GET MORE INFORMATION? 


More information and links to our mailing lists, LinkedIn 
groups, and Facebook group are available at our website: 
http://www.bsdcertification.org 


Registration for upcoming exam events is available at our 
registration website: 
https://register.bsdcertification.org//register/get-a-bsdcg-id 


WHAT’S NEW 


from this BE and boot the system. In addition, the GRUB 
menu will be updated with your current BEs any time you 
perform @beadm create Of beadm destroy Command. 
Since GRUB is performing the initial boot of the system, 
there are a few new commands and configuration options 
available to the end user. First among these is the ability to 
regenerate the /boot/grub/grub.cfg file manually. This file 
is re-created every time you change BEs, but there may be 
cases when you want to manually re-create it, such as af- 
ter placing a new module or setting into /boot/loader.conf. 
To start the re-creation, we will use the grub-mkconfig COM- 
mand: Figure 7. After running this command, any changes 
in the original /boot/grub/grub.cfg Will be lost and replaced 
with an updated configuration file using details about your 
current BEs. However, there may be cases where you want 
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Figure 6. Bootloader 


[roote@pcbsd-8613] ~# grub-mkconfig -o /boot/grub/grub.cfg 
Generating grub.cfg ... 

Found theme: /boot/grub/themes/pcbsd/theme.txt 

done 


Figure 7. Rebuilding grub.cfg 


[root@pchsd-8613] ~# cd /usr/local/etc/grub.d/ 
[rootepcbsd-8613) /usr/local/etc/grub.d# ls 


00 header 10 ktrueos 40 custom README 

10 kfreebsd 30 os-prober 41 custom 

Figure 8. The GRUB configuration script directory 

[root@pcebsd-8613) ~# grep “submenu /boot/grub/grub.cfg | cut -d '"" -f 2 


PC-BSD (default) - 2013-07-19 16:27 
PC-BSD (mewbe) - 2013-07-22 16:31 


Figure 9. Adjusting the /usr/local/etc/default/grub file 


ERUB_THEME=/boot /grub/themes/pcbsd/theme. txt 
GRUB_FONT=/boot /grub/pcbsdfont .pf2 
GRUB_HIDDEN_TIMEOUT=2 
GRUB_HIDDEN_TIMEOUT_QUIET=t rue 

GRUB DEFAULT=saved 


= 
= 
on 
= 
= 


= 


fusr/local/etc/default/grub: unmodified: line 1 


Figure 10. Getting alist of BEs for booting 
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to adjust your boot menu with other options, such as a new 
OS as an additional boot option or changing the default BE 
to boot after the countdown. Let's take a look at how to ac- 
complish these tasks (Figure 8). The first directory of im- 
portance is /usr/local/etc/grub.d. Within this directory is a 
collection of files starting with a numerical prefix. These files 
are shellscripts, run by the grub-mkconfig Command which 
echo out various parts of the resulting grub.cfg file. These 
scripts will normally be replaced when updating the GRUB 
package, however, it is possible to add your own scripts to 
this directory. These new scripts will then be executed along 
with the others and their output added to the end of the grub. 
cfg file created by the grub-mkconfig commana. If you have 
a fairly static group of options, it is also possible to create 
a /boot/grub/custom.cfg file, which will be included at the 
tail end of the grub configuration. Setting the default GRUB 
Boot Environment or OS entry is also easy. To get started, 
we will need to edit the file /usr/local/etc/default/grub 
and add the crus DEFAULT=saved line (Figure 9). 

Anytime we adjust the grub defaults file, we will need 
to re-create the grub configuration file using the com- 
mand Grub Mmkcenmiicu em Moor, onal orub.ctio. After epee 
mkconfig finishes, we can then set the default boot menu 
option with the grub-set-default Command. This com- 
mand will take a numerical argument for the particular 
menu entry to boot, starting with 0. So the first entry will 
be O, the second will be 1, and so on. Using the grep com- 
mand, you can easily see which Boot Environments are 
available for booting. (NOTE: after setting a default Boot 
Environment with the grub-set-default command, if you 
remove a BE you may change the order of the environ- 
ments and you should check your grub.cfg file to make 
sure you are still booting the desired BE) (see Figure 10). 

In this article, we’ve taken a look at how PC-BSD and 
TrueOS use zFs, beadm and cRug to manage backups and 
set the booting of different Boot Environments. In the com- 
ing months, we expect to add new GUI utilities to assist 
desktop users with the management of these technologies, 
and users who want to stay current with these changes are 
encouraged to join us on the PC-BSD Testing & Develop- 
ment mailing lists (http-/ists.pcbsd.org/mailman/stinfo). 


KRIS MOORE 

Kris Moore co-created EasyPBI with Jesse Smith in 2011 and took 
over full development of it for the PC-BSD project in 2012. He lives 
in Tennessee with his wife and two sons and is always looking for 
ways to make computers simpler, but no less powerful, for the av- 
erage user. He is currently employed by iXsystems to work on the 
PC-BSD Project as both a developer and as the manager for the 
PC-BSD PBI repository. He can be reached at: kris@pcbsd.org. 
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They all use Nipper Studio 


to audit their firewalls, switches & routers 


Nipper Studio is an award winning configuration auditing tool which 
analyses vulnerabilities and security weaknesses. You can use our point 
and click interface or automate using scripts. Reports show: 


1) Severity of the Threat & Ease of Resolution 
2) Configuration Change Tracking & Analysis 
3) Potential Solutions including Command Line Fixes to resolve the Issue 


Nipper Studio doesnt produce any network traffic, doesn't need to 
interact directly with devices and can be used in secure environments. 
evaluate for free at 
wuww.titania.com 
CarRputi a en 
Computing 


Security Securi 
pec Awards 


WINNER WINNER Runner-up 


Enterprise Security Network Security SME Security 
Solution of the Year Solution of the Year Solution of the Year 


www.titania.com 
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An email Gateway 


with FreeBSD to Prevent Malware and Undesirable Messages 


Controlling inbound and outbound mail messages is a 
big challenge for sysadmins. Malware can spread quickly, 
infecting dozens of mailboxes and relentless soammers 
send thousands of messages with unsolicited advertising 


and phishing scams. 


What you will learn... 
¢ Basic email gateway setup 


and malware by configuring a FreeBSD email gateway 

system using MailScanner, SpamAssassin, ClamAV 
and Postfix. The email gateway is responsible for analyzing, 
filtering, and cleaning or removing malicious files and mes- 
sages (Figure 1). Figure 2 shows how the mail is scanned by 
the MailScanner. It is delivered to the incoming folder where 
it is analyzed by SpamAssassin and ClamAV and then deliv- 
ered to the quarantine or queue folders. The cleaned mes- 
sages are then delivered to the internal mail server. 


7 his article will show you how to prevent viruses, spam 


Internal Mail Server 


Client 


Figure 1. Basic mail gateway topology 


a BSD 


What you should know... 
¢ FreeBSD shell command line 
« Knowledge of the SMTP, POP and IMAP protocols 


Internet Mail 


MailScanner 

= 

2 

a 

& 
SpamAssassin 


To Mail Server 


Figure 2. MailScanner process flow 
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An email Gateway with FreeBSD to Prevent Malware and Undesirable Messages 


Installing 

Before you install the applications, it is necessary to 
update the FreeBSD ports tree to ensure you use the 
patched versions. 


#cd /usr/ports 
#portsnap fetch update 


Installing Postfix MTA. 


#cd /usr/ports/mail/postfix 
#make install clean 
#cd /usr/local/etc/postfix 


#postalias aliases 


Installing MailScanner. 


MailScanner-4.84.5 3 


Install BitDefender 
Install ClamAVj 
Install ClamAV Module 
Install SpamAssassin 


<Cancel> 


H H 
H | 

| 
H | i 

| 
1 | 

- 
H | 
H a 
i | 
H | 
1 a 
H ry 
H 7 
H | 


Figure 4. Disabling spamd as root 


#cd /usr/ports/mail/mailscanner 


#make install clean 


In the SpamAssassin options, uncheck the AS_- ROOT 
for security reasons (Figure 4). 


Configuring Postfix MTA 

Configure Postfix by editing the /usr/local/etc/postfix/ 
main.cé file (Listing 1), with some basic parameters de- 
scribed below: 


¢ myhostname is the internet hostname of this mail sys- 
tem, use the FQDN; 

¢ mydestination is the list of domains that are delivered 
via the mail delivery transport; 

¢ mynetworks_ style is the method used to generate the 
default value for the mynetworks parameter. The host 
value is when Postfix should “trust” only the local ma- 
chine. 


Create the transport_maps (Listing 2) to set the next 
hop; in our case, that is the internal mail server and 
header_checks (Listing 3) for content inspection of pri- 
mary non-MIME message headers. 


#cd /usr/local/etc/postfix 
#vi transport 

#portmap transport 

#cd /usr/local/etc/postfix 


#vi header checks 


The Sendmail is the default MTA in FreeBSD, so it is 
necessary to disable it and enable Postfix to start at boot 
(Listing 4). 


Listing 1. Basic settings in main.cf 


myhostname = mailgw.acme.local 

mydomain = localhost 

MSI IMieSie reesei) 

mydestination = S$myhostname, localhost.Smydomain, 
liecalhosir 

mynetworks style = host 

mynetworks = 192.168.0.0/24, 127.0.0.0/8 

relay domains = acme.local 

transport maps = hash:/usr/local/etc/postfix/transport 

header checks = regexp:/usr/local/etc/postfix/header_ 


checks 


Listing 2. The /usr/local/etc/postfix/transport file content 
acme locate snep: [LIZ eGo. Un | 


Listing 3. The /usr/local/etc/postfix/header_checks file content 


/*Received:/ HOLD 


Listing 4. Disabling Sendmail and enabling Postfix in the /etc/rc.conf file 


sendmail enable=”"NONE” 
sendmail msp queue enable="NO” 
sendmail outbound _enable="NO” 
sendmail submit _enable=”"NO” 


postfix enable="YES” 
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Listing 5. Configuring Mailscanner 


Sorg-nameS = ACME 


Sweb-site% = www.acme.local 
Run As User = postfix 


Run As Group = postfix 


Incoming Queue Dir = /var/spool/postfix/hold 

Outgoing Queue Dir = /var/spool/postfix/incoming 

MTA = postfix 

Virus Scanners = clamav 

Still Deliver Silent Viruses = yes 

SpamAssassin User State Dir = /var/spool/MailScanner/ 


Spamassassin 


Listing 6. Enabling Mailscanner, ClamAV and SpamAssassin in 
the /etc/rc.conf file 


mailscanner enable="YES” 
spamass milter enable="YES” 


spamd enable="YES” 


clamav _clamd enable="YES” 


Clamav_ milter enable="YES” 


elamay bresie lam enable=" ths ™ 


Listing 7. DNS example 


acme.local. IN MX 10 mailgw.acme.local. 
acme. li@cal. IN A 197 1os.0 2 

mail IN A’ 192.1668.0.2 

matirew UN Abo 2 ee a0 


} 


© junk) & De 
OFf25/2013 04:06 


Me coor pheusiiiacme locals * Reply |=) Forward) Mi Archive 
{Faenamel Fund: Tectet 


Me calexonitacme.local> 7 Other Actc 


Warning: This message has had one or more attachments removed [eicancom, eicar_com. zip) Please read 
the “ACME-Attachment-Warning. txt” attachment(s) for more information. 


Figure 5. Message cleaned 

MailScanner <postmaster@mailgw_acme.local> * Reply | 
Warning: E-mail viruses detected 
Me <morpheus@acme.local> 11 


Our ¢-sail content detector has just been triggered by a sessage you sent: 
To: plexrosaacee. local 
Subject: Fwd: Testez 
Date: Thu Jul 25 16:09:36 2013 


One or gore of the attachments [eicar.com, e1car_com.zip) are o 

the list of unacceptable attachsents for this site and wall nae Tet 
been delivered. 

Consider renaming the files to avoid this constraint. 


The virus detector said this about the message: 


Report: Feport: MailScanner: Executable DOS/aindows programs are dangerous in ¢mai) \eLcar. com 


Figure 6. Alert to the sender 


BSD 


MAGAZINE 


26 


oh The Web 
Postfix official Web Site: http://www.postfix.org/ 

« MailScanner Web Site: http:/www.mailscanner.info/ 

« SpamAssassin Web Site: https:/spamassassin.apache.org/ 

« MailScanner User Guide: http:/www.mailscanner.info/files/ 
MailScanner-Guide.pdf 


Configuring MailScanner 

Configure MailScanner by editing the /usr/local/etc/ 
MailScanner/MailScanner.conf file, to manage the entire 
mail security system. Listing 5 shows the basic param- 
eters necessary to configure it. The parameter Still De- 
liver Silent Viruses should not be used in a production 
environment, because the system will send thousands of 
MailScanner messages to the users. Improve MailScan- 
ner performance by increasing the Max Children parame- 
ter. Keep in mind each process consumes 20 MB. Create 
directory and configure necessary permissions. 


#mkdir -p /var/spool/Mailscanner/incoming && mkdir /var/ 

spool/Mailscanner/quarantine && mkdir /var/spool/ 
Mailscanner/spamassassin 

#chown -R postfix:postfix /var/spool/Mailscanner 


#chmod -R 775 /var/spool/Mailscanner 


Edit the /etc/rc.conf file to enable MailScanner, Cla- 
mAV and SpamAssassin to start at boot (Listing 6). 


Testing 

To test your system, you need to configure the SMTP 
email client to use the email gateway. In my tests, | config- 
ured an MX record pointing to the email gateway (Listing 
7) set in my client. | used the EICAR file to test the mal- 
ware detection and system response. When sending an 
infected file, the system cleans the message, forwards it 
to the recipient (Figure 5) and alerts the sender (Figure 6). 


Conclusion 

The MailScanner is a very powerful security tool and one 
should test several configurations to best understand how 
it can help one’s mail service. MailScanner helps in the 
effective management of your mail security system that 
scales well in larger environments. 


ALEXANDRO SILVA AKA ALEXOS 
Alexandro Silva aka Alexos lives in Salvador, Bahia, Brasil. He 
is an Information Security Consultant at iBliss Seguranga & In- 
teligéncia. He has been using FreeBSD since the 4.11 release and 
can be reached on line at http://alexos.org. 
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Please join us October 25-27, 2013 at the Hyatt in Dulles, Virginia for the first biennial 
vBSDCon event. This exciting weekend will bring together members of the BSD community for a 
series of roundtable discussions, educational sessions, best practice conversations, and exclusive 
networking opportunities. See below for details on this industry weekend not to be missed: 


AGENDA 
- Friday, October 25: Evening Reception 
- Saturday, October 26: General Session, Birds of a Feather Sessions 
¢ Sunday, October 27: General Session, Breakout Sessions 


WHO SHOULD ATTEND 
*Developers «Engineers «Administrators »* Innovators 


TOPICS 
« PkKgNG w/ Baptiste Daroussin 
- A comprehensive look at bsdinstall with Devin Teske 
* Netflix Demo/Presentation with Scott Long 
*netmap with Luigi Rizzo 
- Migration from GCC to LLVM/Clang with David Chisnall 


REGISTRATION INFORMATION 
WILL BE SENT TO YOU IN MAY! 


Questions? Please contact: eventsteam @verisign.com 


_fiin Verisigninc.com 


©2013 VeriSign, Inc. All rights reserved. VERISIGN, the VERISIGN logo, and other trademarks, service marks, and designs are registered or 
unregistered trademarks of VeriSign, Inc. and its subsidiaries in the United States and in foreign countries. All other trademarks are property 
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The Service Spawner 


If you have ever dreamed of software that does user-side 
installation, software configuration, and maintains your 
software automatically without the need for a vast amount 
of UNIX systems knowledge, you should probably take a 


look at the universal Service Soawner. 


What you will learn... 
¢ How software is maintained in the Open Source world. 
¢ How to maintain your applications easily on your servers. 


The story continues and this time, it is about soft- 
ware built on top of architecture used by Sofin — 
The Service Spawner. 

lt is Common in small and medium-sized companies 
where there’s “monit’, or a similar software combined with 
system RC configurations, used to launch all user servic- 
es. In the simple case, this is probably enough, but the 
fun starts when you have dozens of users, each with dif- 
ferent server configuration demands. After a while, you 
end up with tons of manually written configurations and 
scripts, all on the root side, so you then need to perform 
each change manually for each user demand. | could play 
dumb and do some sort of a hack with sudo, but it would 
work only for users who know monit configuration syn- 
tax. How many of your users do that? So | assumed that 
the whole user service configuration should be located on 
the user-side, not the system side. It should be easy and 
straightforward for any user to spawn a new service. So | 
raged, and it happened. 

The SS is a system agnostic mechanism to maintain 
server software, written in C / C++ using the Qt4 frame- 
work (by Trolltech, currently Nokia). It might be considered 
controversial that | used a framework designed for GUI ap- 
plications to write server software, but | consider Qt4 to be 
one of the best cross-platform, high level API's available 
for C++. Besides, TheSS requires only the QtCore part of 
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What you should know... 

« TheSS is based on software built by Sofin. You should know basic 
Sofin principles (BSD Mag from June 2013). 

- If you want to make your own service igniters, you should have (at 
least) basic knowledge of shell scripting and JSON scripting. 


the Qt framework to build (Sofin already provides defini- 
tion with only the core part of Qt without any GUI). That is 
enough for the introduction. Let’s talk about some specifics. 


Design Assumptions 

| started work on TheSS with a simple idea: “! want to 
Spawn any kind of server service, from database servers, 
to web app with ease”. The goal was to create an ultimate 
solution which will provide: 


¢ service installation 

¢ service configuration 

¢ service starting/ restarting / autostarting 

¢ service dependencies support (with explicit order of 
execution) 

¢ service scheduling (built in CRON-compliant asyn- 
chronous scheduler) 

¢ service TCP port checking, random port generation 
for services (including support for static ports) 

¢ automatic service monitoring (including watches on: 
TCP/UDP ports, PID files, process status and custom 
availability checks defined in shell script) 

¢ flexibility (with JSON “software igniters”) 

* easy domain maintenance for services (including 
built-in domain resolving) 

¢ support for expectations and problem notifications 
(low level notification centre) 
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The Service Spawner 


¢ ability to upgrade itself without shutting down running 
services (live upgrades) 

¢ easy to use management panel 

¢ live debugging (log level trigger while running) 

¢ work both for regular users (user services) and super- 
user (root services) 

¢ distribution model in mind 

¢ simplicity in mind (KISS) 

¢ ... and more 


There were many more design assumptions on which 
TheSS was based. | will mention only those most im- 
portant, to help you understand how it works under 
the hood. 

Usually, on standard systems, there are a couple of lo- 
cations where the service configuration might be placed. 
They are /etc/mysoftware, OF ~/.mysoftware. But some- 
times, you will find them also in /usr/local/etc (software 
from FreeBSD ports) or even more fancy places — de- 
pending on the build configuration, operating system and 
so on. TheSS is software that always keeps everything 
— from service configuration, environment, pids, sockets, 
logs and data — in one place, which is ~/softwareData/ 


Mysoft for user and /SystemUsers/SoftwareData/Mysoft 
for root. The structure of “SoftwareData’, for example 
service called “Mysoft” is: 


¢ .../SoftwareData/Mysoft/.autostart — service auto- 
start trigger (TheSS will autostart the service if such 
a file exists) 

e .../SoftwareData/Mysoft/domain — service domain 
(might be set in “igniter”) 

e .../SoftwareData/Mysoft/ports/O — 
port file (might be pool of ports) 

¢ .../SoftwareData/Mysoft/service.conf — service con- 
figuration 

e .../SoftwareData/Mysoft/service.pid — service pid 

e .../SoftwareData/Mysoft/service.sock — service UNIX 
socket file 

e .../SoftwareData/Mysoft/service.env — service envi- 
ronment settings 

e .../SoftwareData/Mysoft/service.log — service log 

e .../SoftwareData/Mysoft/database/ — service database 
directory (only if your service is some kind of database) 

¢ .../SoftwareData/Mysoft/app/ — service application 
data directory (i.e., web application root) 


service default 


www.mtlier.org 
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It is turning the current FHS models inside out, but there 
are more upsides than downsides for this solution. It 
gives you unification and flexibility of configuration for 
each service, which is very important to implement auto- 
matic service management well. 


Service Igniters 
To be flexible enough, all services spawned by TheSS 
have a form of JSON “igniter”. The igniter is just a JSON 
file (with comments support) with defined options for ser- 
vices. By default, TheSS will look for igniters in: 

For regular user: 


¢ /Common/Igniters/Services 
¢ ~/Igniters/Services 


For root: 


¢ /Common/Igniters/Services 
¢ /SystemUsers/Igniters/Services 


Each igniter may implement its own “service hooks’: 


¢ configure — creates service.conf if it doesn't exist 
¢ reconfigure — recreates service.conf, and restarts service 
¢ start — starts service 


Hotkey Window 


6.4. @ 2013 verknowsys.com ServiceSparmne 

tatus: Triggered reconfiguration of opplication: RailsAssets-Rack 
Select service: ngiff Status Flags Autostart 
1 Agins 35981 Running R---- 
38134 Running 
3 TestApp-Nginx 23352 Running 
16209 Running 

47604 Running 

42412 Running 

36792 Running 


“Ese 
R---- 


29138 Running 
46992 Running 
36145 Running 
41474 Running 
18213 Running 
41646 Running 
21398 Running 


Rama aww DD De 
'tnte##& bed i 
obo: 8 @ 4 


Services: Autostart Start slop Restart Validate Install Configure Hew eXplode /=Help 
Cc 


Fistrace Feedebug Fisinfo Fleerror Foerefresh Frenew service Fiedestroy F9=e5S shutdown 
RoilsAssets-Rack - Build Succeed 
=13:54:39 =- Launching serwice: RailsAssets-Redis 

13=13:54: = Launching service: Railséssets=-Puma 
=13:54: = Launching service: RailsAssets-Sidekig 
“13:54:42 - Launching service: RailsAssets-Nginx 

13-13:54:42 - Launching service: RailsAssets-Rack 


Figure 1. Panel — initializing new service 
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| leading /Users/teamon/SoftworeDat 


| 2013-97 - 
| colhost:6379/@ with options {:inamespoce=s>"rails-assets” } 

| 2013-87-13T11:54:212 47669 TID-ovggofwto IMFO: Running in ruby 2.0.0p195 (2813-05-14 r 
| evision 40734) [=£6_64-darwinlz.3.8] 

| 2013-07-13T11:54:217 47669 TID-ovggofvte IMFO: See LICENSE and the LOPL-3.0 for licens 
l ing details. 

|] 2013-07-13T 
|] 2013-07-13T11:54:372 47669 TID-ovggvqnn’ INFO: Shutting down 25 
; Fee FILE REMOVED **"* 
|] 2013-07-13-13h54e41 - 


| loading /Users/teanon/SoftwareDat 


| loading “Users, 
| loading /Us 


| loading /Users/ 


¢ afterStart — is called after service start 

¢ stop — stops service 

¢ afterStop — is called after service stop 

¢ restart — performs stop and start of service 

¢ validate — used by service monitoring mechanism, 
service won't start if this validation fails 

¢ reload — by default sends SIGHUP to service 

e install — installs service if it’s not installed 


Each hook consists of “commands” and “expectations” 
where commands are just plain Shell script, and ex- 
pectations are the way of being notified about hook 
failures. Similar to Sofin, TheSS has Default.json “su- 
per igniter” which is the core of all defined service ig- 
niters. The basic idea was an ability to write igniters 
by hand (simple cases) but also to give support for any 
kind of utility to generate those automatically. One of 
those utilities written in Ruby, by Tymon (teamon) To- 
bolski, is on the way. It is called Hussar and is not re- 
leased yet. 

A major feature of igniters are built in constants. Igniter 
constants are automatically replaced with proper values 
before invoking every hook. Currently there are: 


¢ SERVICE PREFIX — service prefix directory (f.e. ~/ 


SoftwareData/AppName/) 


| evision 40734) [x66_64-darwiniz.3.8] 

| 2013-07-13T11:52:592 47174 TID-ouxpbj07g INFO: See LICENSE and the LGPL-3.6 for licens 
| ing details. 

| 2013-87-13T11:54:182 47174 TID-ouxpbj@7g IMFO: Shutting down 

| 2013-87-13T11:54:182 47174 TID-ouxpiuvec INFO: Shutting down 25 
| **** FILE REMOVED **** 

| 2013-07-13-13 


quiet workers 


hS4e20 - Le 


n/SoftworeData/RoilsAssets-Sidekiq/..//RailsAssets-Rack/service.env 


a/SoftworeDota/RailsAssets-Sidekiq’..//TestApp-ElasticSearch/serwi 


s/teamon/SoftworeData/RailsAss ‘oof /  estApp-Memcached/ service. env 
o/RoilsAss "..//TestApp-Mongodb/ service env 


s/teamon,Softwarebat 5 
/RailsAéssets-Sidekiq/..//TestApp-Rack/service.env 


| RoilsAssets-Rack/ current 


o.lsAssets=-Rack/ current 
iofi2l? 47669 TID-ovggotvto 


5 


IMFO: Booting Sidekiq 2.12.1 using redis:// lo 


11:54:372 47669 TID-ovqgofvto INFO: Shutting down 
quiet workers 


Loading env 
*RoilsAsset 
/RoilsAsset 


g-Sidekiq’..//RoilsAssets-Rock/service.env 


leading /Users/teamon/SoftworeDat s-Sidekig¢..//TestApp-ElasticSearch/servi 
ce.env 

‘RoilsAéAssets 
/RailsAssets-Sidekiq. 


‘reamon/SoftwareData/RailsAssets-Sidekiq/.. 


‘teamon/Softworebote -Sidekiq’..//Testapp-Memcaoched/sservice.env 


s/teanonSoftwareDat _// Tl estApp-Mongouby ce .ony 


//TestApp-Rack/service.env 


| Roilséssets-Rack/ current 


| otlsdssets-Rack/ current 

| 2013-07-13T11:54:422 47908 TID-owlprz218 IMFO: Booting Sidekig 2.12.1 using redis://lo 
| calhost:6379/6 with options {:namespoces>"rails-ossets”} 

|) 2013-87 -=159T11:54:422 47908 T1iD-owlprz2zl8 INFO: Running in ruby 2.@.0pi195 (2013-05-14 r 


evision 40734) [x86 64-daorwiniz.3.6] 


| 2013-87-139T11:54:422 47908 TID-owlprz218 IMFO: See LICENSE and the LGPL-3.8 for licens 
| ing details. 
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The Service Spawner 


Hotkey Window 


Control Ponel ve.¢6.4. © <813 verknowsys.com 
status: Triggered reconfiguration of application: RailsAssets-Rack 
PID Address Status Autostart 


# home Flags 


5§ RoilsAssets-Sidekig 47669 teason-mb. Local :47684 Running 


Services: Autostart Start sTop Restart Validate Install Configure New eXplode f—-Help 
Fistrace Feedebug Foeinfo Fleerror FS=refresh Frenew service Feedestroy FO655 shutdown 


Figure 2. Panel — launching service with dependencies 


Hotke 


Control Panel w.48.4. 6 2013 verknowsys.com 
status: Triggered reconfiguration of application: RailsAssets-Rack 


@ Nome PID Address Status Autostart 


H@ Rai lsAssets-Rack teamon-mb. Local :28448 Running 


Services: Autostart Start slop Restart Validate Install Configure Hew eXplode /=Help 


Fistrace Freedebug Fisinfo Féleerror Foerefresh Frenew service Fiedestroy F9=55 shutdown 


y Window 


evision 40734) (x86_64-dorwiniz.3.8) 

2013-87 -13T11:52:412 46986 TiD-ox4g5varc IMFO: See LICENSE and the LGPL-3.8 for 
ing details. 

2013-87-13T11:52:5627 469286 TiD-ox4g5verc IMFO: Shutting down 
2013-@07-13T11:52:562 465986 TID-ox4g8fhz8 IMFO: Shutting down 25 quiet workers 
eter FILE REMOVED ***" 


loading “Users/teamonsSoftworeDote,sRailsAssets-Sidekig/..//RailsAssets-Rock/service.env 
leading /Users/teamon/SoftworeData/RailsAssets-Sidekig/..//TestApp-ElaosticSearch/servi 
ce. env 

leading /Users/teamon/SoftworeDote/RailsAssets-Sidekig/..//TestApp-Memcochedsservice.env 
leading /Users/steamon/SoftworeDote/RailsAssets-Sidekig/s..//TestApp-Mongodb,service. env 
Leading /Users/teamon/SoftwareDatea/RoilsAssets-Sidekig/..//TestApp-Rack/service.env 
2013-07-13 -139h52e58 - Entering /Users/teamon/SoftwareDota/RailsAssets-Sidekig’, ./ 

Rai lsAssets-Rack/ current 

£013-0/-13-13hoie59 Leaving /Users/teamon/SoftworeData/HaoilsAssets-Sidekig/../R 
ailsAssets-Reck/ current 

2013-07=13T11:52:592 47174 TiD-ouxpb 87g IMFO: Booting Sidekig 2.12.1 using redis://lo 
calhost: 6379/0 ath opt Lons i :nomespace=>"rails-assets"} 

2613-67 -13T11:52:592 47174 TID-ouxph 874 IMFO: Running in ruby 2,.8.8p195 (2013-95-14 r 
evigion 4734") [x 86_64-derwini2.3.8) 

2013-07 -13T11:52:597 47174 TID-ouxpb 7879 IMFO: 
ing details. 
£613-87=13T11:54: 18. 
#6143-87-143T11:54;: 
eee? FILE REMOVED 


See LICENSE and the LGPL-3.6 for licens 
Shutting down 


4 
cs Shutting down #5 quiet workers 


fif4 TID-ouxpb 87g INFO: 
far4 TID-ouxpiuvec IMFO: 


Lead ing /Users/teanon/SoftwareDoata/sRailsAssets-Sidekia’.../RailsAssets-Rock/ service .enw 
leading /Users,teanon/SoftworeData/RailsAssets-Sidekig/..//TestApp-ElasticSearch/servi 
ce.env 

loading /Users/teamon/SoftworeData/RailsAssets-Sidekig/..//TestApp-Memcoched/service.eny 
loading /Users/teomon/SoftworeDota/RailsAssets-Sidekigq/..//TestApp-Mongodb/service.env 
leading /Users/teanon/SoftworeDatasRailsdssets-Sidekig/..//TestApp-Rack/service.env 
e@13-07=13=-13h54e28 =- Entering /Users/teamon/SoftwareDota/RailsAssets-Sidekig/../ 

Rai lséssets-Rock/ current 

2013-87-13 -19h54e?1 - Leaving /Users/teamon/SoftworeData/sRaoi leAssete-Sidekias. ./ 
ailsAssets-Rack/current 

2013-87 =-13T11:54:212 47669 TIiD-owagofvto IMFO: Booting Sidekigq 2.12.1 using redis://lo 
calhost:6379/0 with options {:namespace=>"rails-assets"} 

2013-87 =137T11:54:2127 47669 Tit-ovggotwto IKFO: Running in ruby 2.0.0p195 (2613-05-14 
evision 40/734) |=8b_64-dorwiniz.3.8] 

2013-87-137T11:54:217 47669 TID-ovqgofvto INFO: See LICENSE and the LGPL-3.8 for licens 


ing details. 


=F 


Installing celluloid (8.14.1) 
Installing connection_pool (1.6.8) 
Installing daemons 

Installing eventmac 

Installing j (1.8 

Instal ling x (i. 

Installing 

Installing protection (1.5.0) 
Installing 

Installing redis (3.0.4) 
Installing redis-namespace (1.3.0) 
Installing sidekig (2.12.1) 
Installing tilt (1.4.1) 

Installing sinatra (1.4.3) 
Installing temple (0.6.53 
Installing 

Installing 

Using bundler (1.3.5) 

Your bundle 1s complete! 


Gens in the groups development and test were not installed. 
Tt was installed inte ./bundle. installed 


2013-87=13-14h8rear7 = Leaving /Users/teononsSoftworeDota/Fai lsdssets-Rock/builds? 


Figure 3. Panel - each service information is under a hotkey: “k” (configuration), “L” (log) and “E” (environment) 
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WHAT’S NEW 


¢ PARENT SERVICE PREFIX — prefix of service par- 
ent (filled only if service has parent) 

¢ SERVICE DOMAIN -—- service domain name 

¢ SERVICE ADDRESS -— service address (resolved 
from SERVICE_DOMAIN) 

¢ SERVICE ROOT — service application root (f.e. 
~/Apps/AppName) 


¢ SERVICE VERSION — service version (taken from 
Sofin’s: ~/Apps/AppName/appname.version) 


Please note that igniter constants are values set on run- 
time. Igniter constants work for all hook commands (in- 
cluding the built in cron scheduler) in all igniters. 

The last feature worth mentioning is igniter live updates. 
When you change an igniter, which is used to spawn your 
service, it is automatically reloaded. You do not need to 
restart or reload anything manually to update your sched- 
uler entry. Just edit the igniter, save it, and you are done. 


TheSS in Action 
First of all, TheSS service must be running as a giv- 
en user. Usually, the only thing required is to run two 
commands: sofin get thess and svdss. Before the first 
run, the default igniters must be installed. To do so, run 
ignitersinstall as root (required only once). It will install 
default igniters to /common and /SystemUsers/Igniters. 
After that, TheSS is ready to do its job. If you need your 
own igniters, just put them into ~/Igniters. The panel 
comes up next. To run panel, just run panel. On the first 
run, you will see just lots of empty space which will be 
used later for services list, notifications and service logs. 
To see the panel help, press “?”. To initialize a new ser- 
vice from igniter, hit “N” and type the name. If your ignit- 
ers were installed properly, you should see a scrollable 
list of igniter names there (Figure 1). 

| will just pick one of my testing igniters with depen- 
dencies, based on a Ruby on Rails app. Move the cursor 
with up/down arrows to pick the igniter and press Enter to 
confirm. You will see the initialized service on the list. To 
launch it, press “S” (Figure 2 nad Figure 3). 

lf your igniter is defined properly, you will see your ser- 
vice and its dependencies up and running in a matter of 
seconds. 


Understanding TheSS 
There are several TheSS requirements, all of which must 
be met, to launch your service properly: 


¢ TheSS requires Sofin to be installed on the system. 


Different software build methods are not supported, 
and there is no plan to change that. 
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Glossary 
e FHS - http:/en.wikipedia.org/wiki/Filesystem_Hierarchy_ Standard 
e KISS - An acronym of “Keep It Simple Stupid”. 


e RC - default mechanism of launching services in open 
source systems (rc.d on Linuxes) 


On the Web 
TheSS project page: https://github.com/VerKnowSys/TheSS 
Sofin project page: http://verknowsys.github.io/sofin/ 
My website: http:/dmilith.verknowsys.com/WebLog/Core.html 


¢ Currently, default TheSS shell (used by all igniters) is: 
/Software/Zsh/exports/zsh. If you want TheSS to use 
a different shell, you need to change it in globals.h 
and rebuild the whole software package. 

¢ Service hooks are spawned synchronously, hence ser- 
vice hook commands cannot block execution flow. If they 
do so, you will end up with a blocked execution in a mid- 
dle of some stage. This will look like hang, but it is not. 

¢ TheSS assumes that every igniter is “perfect”. If it is 
not, then your service probably will not work. Please 
take a look at the example igniters bundled with 
TheSS before defining your own. 


Future 

One of the most important upcoming features of TheSS is 
an implementation of the services distribution model. The 
design provides a mechanism of distributed communica- 
tion between cluster nodes, using TheSS to automatically 
move software between machines, scale services with- 
out downtime, and to provide automatically configured do- 
main servers, load balancers and more. 


Summary 

TheSS is actively developed but may be considered sta- 
ble. We have been using it on production servers since 
version 0.22.x (current version at the time of writing this 
article is 0.48.5). Watch for new features! 


DANIEL (DMILITH) DETTLAFF 

Sysadmin of several medium-sized companies in Poland. Enter- 
prise and cloud hater. Currently working at Monterail.com, LLC. 
Working on building self-healing, self-manageable, distributed, 
Al-driven systems. He makes new ideas real while systems are au- 
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FreeBSD Programming 
Primer — Part 7 


In the seventh part of our series on programming, we will continue 
with the menu navigation system and using Javascript. 


What you will learn... 
« How to configure a development environment and write HTML, 
CSS, PHP and SQL code 


represent the three content types that we have de- 


S o far, we have built navigation section buttons that 
fined in content.inc: pages, news and FAQ’s. When 


FreeBsD, 


First FAQ 
Aenean volutpat, ligula vitae laoreet dapibus 


Sars 


imenwie cok DS2013 2544M 


Figure 1. FAQ with Javascript onclick buttons 
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What you should know... 


¢ BSD and general PC administration skills 


the button is pressed, a Javascript popup alerts the user 
as to what button was clicked via the onclick event (Figure 
1). We now need to add additional functionality — when the 
page is loaded, by default the page's links should be dis- 
played, the menu option (or filter) needs to be displayed to 
the user, and when the button is clicked, the menu content 
needs to be rebuilt (Figure 2). Later we will build a more so- 
phisticated menu using the Jquery library. 


Has user chosen 
a menu filter? 


Figure 2. Logic for the navigation menu 
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<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" “http: //www.w3.org/TR/xhtml1/DTD/xhtmll-strict.dtd": 
<html xmlns="http: //www.w3.org/1999/xhtml" xml: Lang="en"> 

<head> 

<meta http-equiv="Content-type" content="text/html; charset='iso-8859-1'" /> 

<link rel="stylesheet" type="text/css" href="/stylesheets/reset.css" /> 

<link rel="stylesheet" type="text/css" href="/stylesheets/global.css" /> 

<script src="/javascript/jquery-1.10.2.min.js" type="text/javascript"></script> 

<script sre="/javascript/jquery.cookie,js" type="text/javascript"></script> 

<script src="/javascript/preload.js" type="text/javascript"> 

</script> 

<title>My first page</title></head><body><div id="page">My first page<div id="debug">&para;</div><div id="h1l 
interdum auctor tellus sed dignissim. Phasellus non orci massa, nec feugiat sem. Vestibulum molestie interdu 
bibendum. Nunc quis elit nulla, sit amet rutrum lorem. Quisque odio est, sagittis nec accumsan ut, placerat : 
amet lectus. Curabitur aliquam dignissim felis, a malesuada leo fringilla at. Sed ornare aliquet lacus, quis 
imperdiet augue mattis eu. Nulla porta odio ut erat consectetur at molestie justo suscipit. Aenean convallis 
pellentesque nisl, vitae posuere mauris facilisis vitae. Morbi in tellus nisl, vel facilisis diam.</div></di' 


Figure 3. Page source showing Javascript Jquery libraries loaded 


Listing 1. postload.js 


// Set navigation menu cookie Smenuvalue = ‘pages’; 
function setnavitem(item) { } 
$.cookie(“navmenuitem”, item); foreach (Scontent tables as Scontenttype) { 
} // Build the option for the content type 
Listing 2. menu.inc SoOLl ome.) <ouLtOm ome lich — sethayitemn. 
<?php Scontenttype.’\’); 
document.location.reload (true) ;">’ .Soffset.’. 
function menu(Stype) { Vouctvsr (Scontent Cype). </burton- ~nbsp,” 
SObESein aa: 
require INCLUDES . ‘content.inc’; 
} 
if (Stype == ‘navigation’) { 
Smenu = ‘/; 


// Build select statement for each content type 


ign Te lieta smenu {= “<div Glass = Menu—- 9 Ssiype i= * > = 
// Omit the UNION keyword on the last item Semis <= <>?  uetrst (ok yoe) al 
Smenuvalue.’)</h2>'; 
Soffset = 1; Smenu .= ‘<p>&nbsp;</p>’; 
pCauedol@es) — COUNT (. Convent tables), Smenu .= Soption; 
Sscivas Vs Smenu .= ‘</div>’; 
Soption = ‘'; 


return Smenu; 


// Get the value of the cookie if set } 


He (tsset(s COOKIE “iavymenuivem? |) 


emenuvalue = $ COOKIE|[ ‘navmenuitem’ |; 


}else { 


BSD |: 


MAGAZINE 


www.bsdmag.org 


ADMIN 


Step 1 - Handling the user interaction 
Ensure you have downloaded the Jquery libraries as de- 
tailed in the previous article. If you view the page source 


First FAQ 


Aenean volutpat, ligula vitae laoreet dapibus 


Navigation (news) 
1.Pages | 2.Faqs | | 3.News | 
Copyngh! ©2013 fob Someniie meiimeniie cok 27/7013 177M 


Figure 4. FAQ page with Javascript and cookie control 


FreeBSD, 


First FAQ 
Aenean volutpal, ligula vitae laoreet dapibus 
Mavigation (pages) 


1.Pages  2.Fags || 3.News 


s Prva ree 


ey < OY cease: HTML C85 iariet DOM Met Cookies * 
7 Cookies Filter Default (Accept cookies) = 
[ Ar 197 beh obs is 86ffon 


Figure 5. Cookie set in Firebug 


<!DOCTYPE html PUBLIC 


<head> 


".//W3C//DTD XHTML 1.0 Strict//EN" 
<html xmlns="http://www.w3.org/1999/xhtml" xml: Lang="en"> 


for any page, it should be similar to (Figure 3). Modify post- 
load.js and menu.inc as follows (Listing 1 — 2). 

lf you now navigate to http://voursiteip/fagq/1, you should 
now see a page similar to (Figure 4). If you click on the 
buttons, instead of a Javascript popup you should see 
the navigation menu title changing to reflect the new se- 
lection. Using Firebug and the Cookie console, you will 
see the content of the cookie changing when a new menu 
item is selected. Deleting the cookie and refreshing the 


Listing 3. add to preload.js 


ji gai ieubler ares 


FUNCE+ On) preimi | Y{ 


‘none’; 


document body.style.drsplay = 


Funct Lon postimaic () { 


S$ (document .body) .fadeIn(500); 


Listing 4. Add to core.inc 


Add just after echo BODY; 


Senos “Seri peel mi io serimr 


Listing 5. Add to core.inc 
hdd just betore ob vend lush) 


SCHOO TSC CIDE pect unnnt) scrip: 


"http: //www.w3.org/TR/xhtml1/0TD/xhtmll-strict.dtd"> 


<meta http-equiv="Content-type" content="text/html; charset='iso-8859-1'" /> 
<Link rel="stylesheet" type="text/css" href="/stylesheets/reset.css" /> 

<Link rel="stylesheet" type="text/css" href="/stylesheets/global.css" /> 
<script src="/javascript/jquery- 1.10.2.min.js" type="text/javascript"></script> 


<script src=" 


" type="text/javascript"></script> 


<script src="/javascript/preload.is" type="text/javascript"> 


</script> 


<title>FAQ 1</title></head><body><div id="faq"><div id="heading" 


class="heading-1">First FAQ</div><div 


id="content" class="content-1">Aenean volutpat, ligula vitae laoreet dapibus</div><div class ="menu-navigation"> 


<h2>Navigation (pages)</h2><p>&nbsp;</p><button onclick="setnavitem('pages'); 


document. Location. reload(true);">l. 


Pages</button> &nbsp;<button onclick="setnavitem('faqs'); document. location. reload(true);">2. Faqs</button> &nbsp; 


<button onclick="setnavitem( 'news'); 


document. location. reload(true);">3. News</button> &nbsp;</div></div><div 


id="Licence"><a href="Licence.txt" title="Copyright and Licence details">Copyright &copy; 2013 Rob Somerville 
me@merville.co.uk</a></div><script src="/javascript/postload.js” type="text/javascript"></script></body></html> 


Figure 6. Page source showing button options 
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Listing 6. Add to mysql.inc 


Returns an array of rows or NULL on no result 
FUNC eLOn Mysql ererch rowel scl )an 
// Returns an array of rows or NULL if no result 


Sdb = new mysgli(DBSERVER, DBUSER, DBPASSWORD, 
CMSDB) ; 


Te [db >> Commecercr rao > 0) 4 
die(‘Unable to connect to database ['‘ 


SCI) e ewe ec Sisieie ||! |) & 


if ('Sresult = $db->query($sql)) { 
if (DEBUG) { 
die(‘There was an error running the query [* 
Sdb=remrone.. 4) 9); 
} else { 
die(*"); 


while (Srow = $Sresult->fetch row()) { 
ois) Ulle) Steels - 

// Free the result 

Sresult->free (); 

// Close the connect om 

Sdb->close(); 

if (isset(Sr)) { 
return Sr; 


} else { 
return NULL; 


Listing 7. Add to core.inc 


function arraytolinks (Smysqlfetchrows) { 
requilre INCKUDES — = “coOmeentownc. 


// Convert a MySQL result set into a set of links 


// Requires ID (page id), title and contenttype 


Simkceu= “di telass— menulimaka = 


clainikse = <ul, 
if (Smysqlfetchrows) { 
foreach (Smysqlfetchrows as Skey => Svalue) { 
// Convert the content type to the relevant 
table name. 


7) (See sCOntent ine 


Celta = aicigely Sealed (Ovals ||” ||, Seemesac 


tables); 


Linke.) “Glirca ner“ /" -soath. 7 i ovale [Olin 
title="' .Svalue[1].’”>’. 


Svaliie [ili —/ a>] ie. 


} 
Satanic: tee) le 
Slinks 2= </div>"; 
}else{ 
Slanks = “<lisSorry = mo content available</ 
baa he <7 cave = 
} 


return Slinks; 
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Listing 8. Full listing of menu.inc 

<?php 

function menu(Stype) { 
require INCLUDES NCONGenE nC: 7 


if ($type == ‘navigation’) { 


SOnrsciue— ils 


7 Cavedorics) — COUNE (convene, eables), 
-sql = 9"; 
Sop elon = "> 


di (Gee one Welle © iis cOelaS ane Sac 


if (isset($ COOKIE[‘navmenuitem’ ]) ) { 
emenuvalue = 5 COOKIE[ ‘navmenuitem’ |; 
jelse{ 
Smenuvalue = ‘pages’; 


foreach (Scontent tables as Scontenttype) { 


7/7 Bovid Ehe option fOr ene Content type 


Soption .= ‘<button onclick="setnavitem(\’’. 


Scontenttype.’\’); 


document: locaton, ceload (Erue) ss -sofrset.”. 


Vuchest (Scomtenttype). </button> enosp, 
SOLESeE tt, 


// Build the SOL statement for the menu item selected 


Ssql = “SELECT id,title,’”.Smenuvalue.”’ AS 


contenttype FROM “.Smenuvalue.” 


WHERE stetus = 72 ORDER BY title;”: 


// Get the result 

~Lesult = mysql fetchrows (osql); 

// Convert the array into HTML links 
Slinks = 


arraytolinks (Sresult) ; 


SlSigh — s 


Smenu .= ‘<div class =”"menu-’ Stype ee 
Sens .—= <“iZ> “2 UCtibnsE(=lype) = = 1. 
Smenuvalue.’) - ‘.Scategories.’ 

categories</h2>’; 

SmenU s=) “<p> enbsp,</ p> 3 

Smenu .= Soption; 

Smenu .= Slinks; 

Smenu .= ‘</div>’; 

return Smenu; 


Listing 9. Changes to faq_tempate.inc 
render (Stheme[ ‘heading’ ]) ; 
render (menu ( ‘navigation’ )); 


render (Stheme[ ‘content’ ]); 


Listing 10. Modify global.css 


.menu-navigation { 
#E5E6AD; 
border: lpx solid #DADADA; 


background=co lor: 
Padding: 10px; 
Hogi: Lignin. 
iileniele ligt alkene) ef ellen. 


Mac Ti=DOk bom: hUiox- 


#news, #page, #faq { 
border: lpx solid #DADADA; 
ienalea ol eejeys JL e\Ohep.<k 
padding: 32 0px; 
min-height: 640px; 


overflow: auto; 


Listing 11. Add global menu support to News, FAQ and pages 
templates 


Add at the beginning of each file (e.g. just before 
render (Stheme[ ‘heading’ ]) ;) 


render (menu(‘global’)); 


Listing 12. Add to preload.js 


function globalmenu() { 


Ss(function() {S( “menu” ).menu();})-; 


y 
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Listing 13. header.inc 


Mic ty / WWW wo. Org, TRY xhimil/ DED) xnimil=siieiok adidas > 


<head> 


</script> 


<!DOCTYPE html PUBLIC “=//W3C//DID XHTML 1.0 Strict//EN” 


<html xmlns="http://www.w3.org/1999/xhtml” xml:lang=”"en”> 


<meta http-equiv="Content-type” content="text/html; charset=’iso-8859-1'” /> 
<link rel="stylesheet” type="text/css” href=”"/stylesheets/reset.css” /> 

<link rel="stylesheet” type="text/css” href=”"”/stylesheets/global.css” /> 

<link rel="stylesheet” type="text/css” href="”/stylesheets/jquery-ui.css” /> 
<script srce=”"/javascript/jquery-1.10.2.min.js” type="text/javascript”></script> 
<script src=”"/javascript/jquery.cookie.js” type="text/javascript”></script> 
<script srce=”"/javascript/jquery-ui.min.js” type="text/javascript”></script> 


<script src=”"/javascript/preload.js” type="text/javascript”> 


page should load the default menu type of Pages (Figure 
5). The titles have also been cleaned up using the PHP 
ucfirst() function call to uppercase the first character 
of the selection, and we have added a sequential option 
number to each menu item. 

One disadvantage of this method is the following piece 
of code as shown in (Figure 6). Each button has two piec- 
es of Javascript attached, setnavitem() and document. 
location.reload(). The former sets the cookie via our 
function call in postload.js (and subsequently via the 
jquery.cookie.js script) and then refreshes the page. This 
causes the page to flicker every so often when the con- 
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(Hic Pages 
My fret page 
age header News 


Lorem ipsum doll pages 
Phasellus mon orci massa, nec feugiat sem, Vestibulum molesie interdum bibendum, Mune quis oli 


ecletur adipiscing elit, Mauris interdum auctor tellus sed dignissim. 


nulla, sit amet ruirum lorem. Quisque odio est, sagittis nec accumsan ul, placerat sit amet lectus 


Figure 7. FAQ page menu 
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Third FAQ 
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam eros | Mavigation (pages) -Jcalegodes i 
nibh, dapibus sed suscipi mec, sollicrudin congqué ante. Nulla lacwia 1 =H ITY 


& Pages i 2. Fags | 1. News it 
ullamcorper tristique. Nam id malesuada arcu. Pellentesque diam eros, oy... a 


Warns al consequal Sit amet, Blainat Ul neque Danec lempor cagnissim | 
lacus, sit amet faucibus leo. In commode omare sem, non euismod — 


nunc alguet solicitudin, Sed solicitudin augue al lacinia tempor. Curabitur Egula elit, vestibulum sit 


Figure 8. FAQ fags menu 
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tent is reloaded. A better way of implementing this would 
be to use Ajax, but for the time being, we will demonstrate 
a useful Jquery call — Fade in. 

Add the following code to preload.js (Listing 3) and core. 
inc (Listing 4 and Listing 5). 

This will halt the display of the page, allow the menu to 
be built etc. and the page will then fade in. The time can 


)) FreeBSD, 


Third FAQ 


Lorem ipsum doler sq amet, consectetur adipiscing alt, Aliquam eras - Mavigation (lags) - 3 categories 
nibh, dapibus sed suscipit nec, sollicttudin congue ante. Nulla lacinia + Faas 


| 1 Pages 1 2. Fags | News i 
ullameorper tristique. Nam id malesuada arcu. Pellentesque diam eros, | $5.5, = aA 
Vanus al consequat sil amet, blandit ul neque. Donec lempor dignissim ass 
lacus, sit amet faucibus leo. In commoda omare sem, non euismod 
nunc aliquet solicitudin. Sed sollicitudin augue at lacinia tempor. 49 


Curabtur ligula eft, vestibulum sit amet lacus vitae, cursus rufrum | 


sapien. Aliquam elementum, augue a sodales venenalis, odio mi a) 
lempus ipsum, congue gravida turpls est eu sapien. Nam viverra turpis ee 
non risus auctor vehicula. Etiam nibh diam, interdum non ultricies a, - 
dapibus vel purus, Aliquam convalis interdum magna, Curabdur vilae lobortis massa, Nam pulvinar sed 
diam in adipiscing. Etiam ac lectus at purus porta vulputate. Integer convallis volutpal odio, eu lobortis 


Figure 9. FAQ news menu 


Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam @roS Havigation (news) -: 

Y . A 

nibh, dapibus sed suscipit nec, sollicitudin congue ante. Nulla lacinia — 
ullamcorper tristique. Nam id malesuada arcu. Pellentesque diam eros, , «. 

4 Pst 


VArIUS al consequal sl amel, blandd ul neque. Donec tempor dignissim _ 
lacus, sit amel faucibus leo. In commodo amare sem, non evuemod 
nunc aliquet sollicitudin. Sed sollicitudin augue at lacinia tempor. Curabitur ligula elit, vestibulum sit 


amet lacus vilaé, CUrSUS rulrTum Sapien. Aliquam élemeénium, augue a Sodales venénalis, odio mi 


Figure 10. Jquery multi-level menu 
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be adjusted by incrementing or decrementing the fadern() 
parameter. While this is not an ideal solution, it does dem- 
onstrate the ease of integrating Jquery with a web page. 


Step 2 - Displaying the links 
Now we need to plug in the SQL result to our menu mod- 
ule. Add the following code (Listing 6-8). 

We now need to make a few minor modifications at the 
theme and CSS levels, so change faq_template.inc to dis- 
play the menu before the content (Listing 9). 


Listing 14. Additions to menu.inc 


Add elseif at the end of the navigation block 


Smenu .= ‘<div class ="menu-’ . Stype . ‘”>’; 

Smenu .= ‘<h2>’ . ucfirst(Stype) . ‘ (*. 
smenuvalue.’) — ‘,.Scategories.’ 

categories</h2>’ ; 

Sem = 9 ““p>anbso, <7 pe: 

Smenu .= Soption; 

Smenu .= Slinks; 

Smeniec= <4 dive. 


return Smenu; 


Jelseif (Stype == “global”) { 


?> 


<ul id="menu”> 
<li><a href="/">Home</a> 
<ul> 
<li><a href="/page/1”>Pages</a></1i> 
<li><a href="/news/1”>News</a></1i> 
<li><a href=”"/faq/1”>FAQ’ s</a></1li> 
</ul> 
</li> 
</ul> 


<?php 


Listing 15. Add to global.css 


-Ui-menu { 


Wacicn: 1 p0ps- 
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Useful links 


Jquery UI source — http:///queryui.com/resources/download/ 
Jquery-ui-1.10.3.Zip 
Jquery menu reference — http:///queryui.com/menu 


This will float the navigation menu on the FAQ page to 
the right and increase the height of our news, page, and 
FAQ content to accommodate the new menu. 

See (Figure 7-9) for the final result. | added an extra “Ip- 
sum Lorem” to pad the content out in FAQ 3. Note how the 
menu responds to user input decoupled from the content 
that the user is currently visiting. 


Step 3 - Global website menu 

Jquery provides an extensive library for the user interface. 
Rather than building the Javascript and CSS from scratch, 
we can install the CSS and JS libraries quickly into 
our CMS. 

Download Jquery-ui-1.10.3.zip and extract Jquery-ui. 
css into the stylesheets directory and Jquery-ui.min.js into 
the javascript directory. Use MC, or extract the file into a 
temporary area using unzip. 

Add the global menu to all of our content templates (news _ 
templates.inc, pages template.inc and fags tempate. inc) 
and add the Javascript function to preload.js. Add the Ja- 
vascript and CSS files to the header.inc file and add a new 
menu option to menu.inc and finally tweak our CSS file to 
reduce the width of the menu (Listing 11-15). 

Finally, visit the homepage of your site with your browser, 
refresh the page and voila, one multi-level menu (Figure 10). 


In the next part 
We will continue refining the menu system and start build- 
ing the user interface. 


ROB SOMERVILLE 

Rob Somerville has been passionate about technology since his 
early teens. A keen advocate of open systems since the mid-eight- 
ies, he has worked in many corporate sectors including finance, 
automotive, airlines, government and media in a variety of roles 
from technical support, system administrator, developer, systems 
integrator and IT manager. He has moved on from CP/M and nixie 
tubes but keeps a soldering iron handy just in case. 
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PKGNG 


The future of packages on FreeBSD and PC-BSD 


This article will show how to install, upgrade and remove 
packages using pkgng. It will also discuss some of the 
improvements over pkg_*tools and demonstrate how 
okgng will benefit end users. In addition, the upcoming 
functionality of pkgng will also be briefly discussed. Finally, 
it will show how to install Gnome 3 and Cinnamon on 
PC-BSD Rolling Release using Pkgdemon. 


What you will learn... 

« How to adda package site; 

« How to install and upgrade packages; 

- How to remove packages and all of the dependencies that were 
installed by the packages. 


ll of you want to know how to use a third party 
pane repository. This article will use Pkgde- 

mon with the PC-BSD Rolling Release as an ex- 
ample to demonstrate the process. This article will show 
the process of adding a package site and then installing 
packages on the PC-BSD Rolling Release. The same in- 
formation can be applied to a fresh install of FreeBSD 
9.1 onward, or a FreeBSD 9.1 installation where pkgng 
was used to install all previously installed packages. It 
will cover basic maintenance tasks. It will also demon- 
strate the advantages of using the new package system 
as well as its future potential. 


Before PKGNG 

Users of FreeBSD and derivative systems have always 
had primarily two ways of installing software. The first 
one is through the use of ports and building binaries from 
source code. The second one is by using packages. Any- 
one who has ever installed packages before on FreeBSD, 
will be familiar with the problems associated with using 
pkg *tools. This may be one of the many reasons most 
users choose to stick with ports. While installation of soft- 
ware by ports is often ideal for servers and advanced us- 
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What you should know... 
¢ Basic shell commands; 
¢ How to install PC-BSD Rolling Release. 


ers, it is not often ideal for a user who wants or needs a 
desktop that works out of the box. 

A good example for me is when | need to provision a 
new machine for work purposes. | need to be able to get it 
installed quickly, so | can begin to look up customers and 
troubleshoot issues. | don’t have time to wait around for a 
couple of days or have a half-usable system until all of the 
software is installed. So for me, installing from packages 
on a desktop is a perfect solution. 

At my workplace, we have used FreeBSD on our serv- 
ers for years and we love it for its rock-solid stability, secu- 
rity, and portability. A few of us experimented with the idea 
of using FreeBSD as a desktop a few times at work. How- 
ever, it became clear that as much as we loved FreeB- 
SD, none of us could afford the downtime. Then PC-BSD 
came along with a few really solid releases, starting with 
8.0, and that really began to meet our needs. 

PC-BSD has, of course, solved many of the problems 
associated with using packages by developing a third 
method called PBI. This self-contained method allows 
quick installation of many desktops and even certain serv- 
er applications. It has allowed me and others to use Free- 
BSD at work on our desktops. 
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However, PC-BSD has also relied on using packag- 
es for its desktop interfaces quite often. The shortcom- 
ings of pkg “tools are less noticeable, but nonetheless 
are still noticeable, when it comes to exceptionally long 
installation times and software upgrades. It would be- 
come even more noticeable if a user were to try to in- 
stall software themselves through packages. Package 
conflicts could occur and a user could easily break their 
entire desktop. 

For example, to upgrade from PC-BSD 9.0 release to 
9.1, release packages would have to be manually delet- 
ed behind the scenes. In addition, the PC-BSD operating 
system had to know what packages a user had installed. 
lf a user had installed their own packages or ports, they 
would have to be removed behind the scenes. 

lf the PC-BSD installer had not done this, the entire 
desktop could be broken. As a result, if a user has addi- 
tional ports or packages that are not part of the PC-BSD 
installation, they will be deleted during an upgrade. This 
was not a huge issue, it was just an obvious limitation of 
pkg_ “tools. 


pkg_*tools and third party repositories 

To give further examples of pkg “tools issues, | will use 
my website: Pkgdemon. Pkgdemon is a 3rd party pack- 
age repository for FreeBSD, GhostBSD, and PC-BSD. 
The purpose of Pkgdemon right now is to provide ear- 
ly access to easily-installable Gnome 3 and Cinnamon 
packages. 

When | started Pkgdemon, these ports were not yet 
available in the ports tree, and | wanted to use them at 
work on my PC-BSD desktop. Knowing that | couldn't af- 
ford to have a lot of downtime at work, | devised a way to 
make the installation easy enough so | could install it on 
my work system. 

After compiling ports into packages, there was one obvi- 
ous problem. Installing Gnome 3 from using pkg_ “tools is 
not easy if you have other packages installed which con- 
flict with Gnome 3. Even if you know what the names of 
the packages are, it is still not easy. For example, for each 
pkg, you would have to list the entire name of the pack- 
age. You are also likely to have to force the package to 
uninstall. Here is an example of one (out of the hundred) 
pkg delete Commands required for installation on PC- 
BSD 9.1 Isotope. 


pkg delete =f py27-dbus-0.84.0 
You may be thinking that | could have recursively delet- 


ed Gnome 2 and all of its dependencies. However, this 
would break both my installation and the Nvidia driv- 
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ers, as deleting Gnome 2 recursively would have unin- 
stalled components required by Nvidia and Xorg. This 
was not an option for this script. This situation was 
somewhat eased later by pkgng as you will see later in 
this article. 

| had to create executable shell scripts to remove the 
several hundreds of packages. In addition, occasion- 
ally other packages that were outside of my Gnome 3 
repository were more recent versions than what Gnome 
3 expected. For example, Gnome 3 expected that libp- 
cre.so.1 and libpcre.so.3 were installed. | had to man- 
ually create symbolic links within the script. | had to 
do this for a few other libraries as well otherwise the 
package installation would fail and several applications 
would not open. This is no longer necessary as often, 
with pkgng. 

In addition, | wanted to be able to host other desk- 
tops for installation besides Gnome 3. For now, the ad- 
ditional desktop is Cinnamon. In the future, | want to 
add Mate, possibly CDE, and some others. To make 
that easy, | had to create an interactive menu within the 
shell script to allow the user to choose which combina- 
tion of desktops to install. This is also no longer neces- 
sary with pkgng. 

When | had made changes to the ports tree, | forked 
Gnome 3 for packaging purposes. | needed to devise a 
way to give myself and others an upgrade path. This in- 
volved adding a list of packages that were installed by 
Gnome 3 or Cinnamon and making sure those were re- 
moved as well. If | had forgotten a package it would be- 
come obvious as the script would fail during installation. 
This was also drastically eased by pkgng. 

Listing 1 is what the script would look like in a terminal. 
Without this script, users would have to manually remove 
hundreds of packages by hand in order to install Gnome 3 
using packages created by pkg_ “tools. The source code 
for this script showing the commands can be viewed here: 
https://github.com/pkgdemon/freebsd-pkgdemon/blob/ 
master/PCBSD9. 1-x64-pkgdemon-v1.01.sh 

lf you need the x86 version of the script, please go to 
the Pkgdemon github url: http:/Avww.github.com/pkgde- 
mon/treebsd-pkgdemon 


PKGNG 

PKGNG is referred to as the next generation package 
manager. It uses a SQLite database to store all of the info 
about the installed packages on your system. The location 
of the database still resides in /var/db/pkg, but now as a 
single file. This is in contrast to otherwise what would be 
many folders and files containing information about the 
installed packages. 
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Storing the information about packages in a database 
makes searching for information about installed packages 
fast. It also helps make installation and removal fast, and 
it allows for easier upgrades. 

| could go on about the technical details of what makes 
pkgng so great, however, nothing will show it as well as 
using it. To do that, | must introduce PC-BSD rolling re- 
lease and try to explain briefly what it is. 


PC-BSD Rolling Release 

Some time ago, Kris Moore announced that PC-BSD 
would be adding a rolling release model in addition to 
the current stable releases. The current stable release is 
PC-BSD 9.1 Isotope. As part of this release announce- 
ment, it was mentioned that PC-BSD would begin using 
pkgng, and that all of the packages would be converted 
to pkgng. 

In addition to this, it was mentioned that a pkgng re- 
pository would be created for upgrades of system pack- 
ages. This meant that if a new version of KDE or a new 
version of Xorg were released into FreeBSD ports that, if 
you were running the rolling release, you could upgrade 
to them as they periodically rebuild the packages for the 
repository. That is as opposed to having to wait for the 


next stable PC-BSD release. This currently only happens 
about every year or so when a new release of FreeBSD 
occurs. To get a rough idea of what types of commands, 
how many packages might have to be removed, and what 
might be required to upgrade to a new version of KDE 
or Xorg using pkg-tools, please take a look at script from 
Listing 1. With pkgng, it is just one command. 


pkg upgrade 


lf that isn’t a huge reason to switch to pkgng over 
pkg_ “tools right now, what is? 


Adding a third party repository 

Now | am going to show the installation process for Pk- 
gdemon using PC-BSD Rolling Release. | was told that 
eventually there will be a special page on the PC-BSD 
website for the Rolling Release with download links. A few 
download links will be included at the end of the article for 
Rolling Release. 

Before following this guide please remove XFCE, LX- 
DE, and Gnome. If you are doing a fresh install, do not se- 
lect either of these or the following commands will remove 
these packages. 


Listing 1. Script’s GUI 


PCBSD9. 1—-x64-pkqdemon—v 1201 lsh 
[xcor@psBSDSOZ0) 7 vcs 27 PCBSD951—x64-pkoderion—y W017 sh 


what you are doing. 


I am not afilliated with FreeBSD, or PCBSD. 


He HEHE HEHE HE 


# Gnome 3 Installer for PCBSD 9.1 Isotope Edition 


Press enter to continue or control ¢c to abort 


[root@psBSD8920] “# fetch http://www.pkgdemon.com/downloads/scripts/PCBSD9.1-x64-pkgdemon-v1.01.sh 


W003 or 4 = is 459 kBps 


Ht HH EEE HE HH EEE HHH EEE HERE EE EEE EE EE HE EE EH EEE EEE EE EEA EE HE EE EH HH EEE HE EEE EE HEHEHE 
By installing this software you are in agreement that I am not in 
anyway responsbile for what may happen to your computer. 


Please do not run this script unless you have made a backup and know 


Please GO NOe COmeaces ENE FOr SUppPOLrE Br VOU rum cals SerupeE,. 


at ae Ae Ae Ae ae ae ae aa ae ae AE AE AE aE aE AE ea aaa aE aE Ea aE aa aaa 


at ae ae Ae ae ae ae ae ae aa ae ae AE AE aE aE AEE aaa aE EE aa aaa Ea 


at ae ae Ae ae ae ae ae ae aaa AE AE aE aE EE aaa aa aE Ea aE aaa Ea 


KKEKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK 


Phase 1 will download aprroximatley 58lmb of packages from Pkgdemon 


KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK 


SH HEHE HEHE HE 


it 
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pc-metapkgmanager del GNOME 
pce-metapkgmanager del GNOME-Accessibility 
pc-metapkgmanager del GNOME-Games 


pc-metapkgmanager del GNOME-Net 


pce-metapkgmanager del GNOME-Utilities 
pce-metapkgmanager del XFCE 


pce-metapkgmanager del XFCE-Plugins 


pce-metapkgmanager del LXDE 


pkg autoremove 


The pc-metapkgmanager commands are specific to 
PC-BSD; however, the pkg autoremove Command is spe- 
cific to pkgng. In this example, pc-metapkgmanager will 
only remove a single package, not all of its dependen- 
cies. Executing pkg autoremove will remove all of the 
dependencies required by Gnome 2, XFCE, and LXDE. 
Before, this would have also removed things required for 
the Nvidia driver and Xorg. This is no longer the case 
with pkgng, unless those packages are also removed be- 
fore executing pkg autoremove. 

After PC-BSD is installed, it is recommended you login 
to either the Fluxbox or KDE window manager and open 
a terminal. The first step will be to add the pkg repo key, 
which is used for verification. 

To use the package signing key you will need to fetch it 
Into /usr/local/etc/. For example: 


ed /usr/ local/etc 
fetch http://www.pkgdemon.com/downloads/pkgdemon.cert 


After this, you will need to edit /usr/local/etc/pkg.conf 
and comment the following lines like this example. 


#packagesite: http://pkg.cdn.pcbsd.org/9.1-RELEASE/amd64 
#HTTP MIRROR: http 

#PUBKEY: /usr/local/etc/pkg-pubkey.cert 

Add the public cert key for pkgdemon tO pkg.conf. 

PUBKEY: /usr/local/etc/pkgdemon.cert 

Then, add the appropriate line below to pkg.conf, de- 
pending on which architecture you used to install PC- 
BSD rolling release. PCBSD Rolling Releases are now 


only available for 64 bit systems. 
64 bit 


packagesite: http://www.pkgdemon.com/freebsd:9:x86:64/latest 


There are still a few packages, however, that will have to 
be removed manually as the versions of these packages 
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cannot be removed automatically by pkgng. The following 
packages are not yet built to be upgrade aware. Therefore 
they must be removed manually first. The -f flag is used to 
force removal. If -f is not specified, the packages will not 
remove as they are required by pcbsd-base. 


pkg delete -fy at-spi 

pkg delete -fy zenity 

pkg delete -fy gnome-session 

pkg delete -fy metacity 

pkg delete -fy gdm 

pkg delete -fy evolution-data-server 
pkg delete -fy gnome-desktop 

pkg delete -fy gnome-panel 

pkg delete -fy gnome-settings-daemon 
pkg delete -fy libgnomekbd 

pkg delete -fy gnome-power-manager 
pkg delete -fy gnome-keyring 

pkg delete -fy libgweather 

pkg delete -fy gnome-menus 

pkg delete -fy libwnck 


pkg delete -fy gnome-control-center 


pkg delete -fy farsight2 


The -y flag will automatically confirm the removal of 
each package so that you are not prompted to type y to 
remove each package. 

You should notice a couple of things here. One is that 
the list of packages to delete is dramatically smaller than 
it would be with pkg_“tools. In fact, hundreds of packag- 
es less than when the script was removing the conflicting 
desktops for you and there were still hundreds of packag- 
es to remove. The second thing you should notice is that 
when you type pkg delete, you no longer have to include 
the exact version. 


Upgrade packages 
pkg upgrade -fy 


It is safer to use -f flag here to force the upgrade of these 
packages. This will force an upgrade of the entire pack- 
age set to ensure compatibility with Gnome 3. Other- 
wise certain packages necessary for this example may 
not successfully install and Gnome 3 will crash when 
launching. The -y flag is used again to automatically 
confirm to make installation easier. 


Install Packages 


After the packages have been upgraded, run the following 
command to install Gnome 3. 
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pkg install gnome3 
Installing Cinnamon is also fairly straightforward. 
pkg install cinnamon 


You may notice that | am not going to tell you to create 
a symbolic link with my rolling release instructions. Pk- 
gng is smart enough to figure out what version of pcre 
Gnome 3.4 wants. 


Package removal 

The way package removal works with pkgng is much bet- 
ter. If you wanted to remove my Gnome 3 and Cinnamon 
packages and go back to using what’s provided by PC- 
BSD, it’s fairly easy to do. 


pkg remove gnome3 
pkg autoremove 
pkg remove cinnamon 


pkg autoremove 


Then just set your packagesite and cert file back to what 
was provided by PC-BSD. If you just commented the 
lines, you can uncomment them. 


packagesite: http://pkg.cdn.pcbsd.org/9.1-RELEASE/amd64 
HITE MIRROR? Leip 
PUBKEY: /usr/local/etc/pkg-pubkey.cert 


Now comment or delete the lines for Pkgdemon. 


#packagesite: http://www.pkgdemon.com/freebsd: 9:x86:64/latest 
#PUBKEY: /usr/local/etc/pkgdemon.cert 


pkg upgrade -fy 


On The Web 

- PC-BSD Official FTP Site for Rolling Release: ftp://ftp.pcbsd. 
org/pub/mirror/9.1-RELEASE/ 

« Pkgdemon official Web Site: http:/www.pkgdemon.com 

« Pkgdemon Guide for Rolling Release: http:/www.pkgde- 
mon.com/support/install-pcbsd-testing 

« Pkgdemon Github to view scripts: http:/,www.github.com/ 
pkgdemon/freebsd-pkgdemon 

- Dedicated server hosting for Pkgdemon provided by Sum- 
ner Communications: http://www.sutv.com 


Future pkgng features 

It has been announced that pkgng will soon be getting 
some additional features, not entirely covered here. It es- 
sentially provides a solid support to a graphical user inter- 
face so that installing packages can become available for 
starters. 

Currently, using my installation guide and the rolling re- 
lease, you will have to replace PC-BSD’s repo with my 
repo in order to install packages from Pkgdemon. An up- 
coming feature will allow upgrading across multiple repos- 
itories. This means you should be able to reference both 
PC-BSD’s repository and my repository to get the latest 
packages from both Pkgdemon and PC-BSD. 


Conclusion 

Pkgng is a 21st century package system for FreeBSD. 
It will soon allow using multiple repositories concurrent- 
ly for upgrading and software installation. It will prove to 
be beneficial for end users by providing easy installation 
and upgrades of software for FreeBSD using the pack- 
ages system. 


JOE MALONEY 

Joe Maloney lives in the United States with his family. He works 
as an Assistant Network Administrator for an Internet Service 
Provider at Sumner Communications. He likes to hang out on IRC 
on the Freenode network in the #PCBSD, GhostBSD, and FreeB- 
SD-Gnome channels as malco_2001. 

He can be reached online at http://www.pkgdemon.com, pkgde- 
monteam@gmail.com, or by IRC. 
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Faster. 
Better. 


Reliable. 
Trusted by over 500 ISPs worldwide. 


Hyper is the first multimedia cache fully developed in Brazil, by Taghos. 
With Hyper, ISPs can save on network Dandwidth while increasing 
content-delivery speeds, resulting in end-customer satisfaction. 


Features: 

- 24x7X365 always-on support 

- Active monitoring 

- Automatic updates 

- Appliance or license 

- Easy deployment 

- Configuration and reports via 
web interface 


Cache 
Up to 15 Mbps 8 G 1x 11B 
Up to 50 Mbps 2x 11B : 
Up to 100 Mbps : 2x 11B 1x 160 GB 
Up to 150Mbps 3x 2 TB 1x 160 GB 
Up to 300 Mbps B 5x 2 1B 1x 240 GB 
Up to 500 Mbps | 7X 2 TB 1x 480 GB 
| Up to 1 Gbps 4 GB 10x 1 TB 1x 480 GB 
Remote Instal| Up to 2 Gbps 24x11B 3x 480 GB 


13000 Up to 3 Gbps 128 GB 32x 1 1B 5x 480 GB 


Using your hardware 


Visit us at WWw.taghos.com and start saving bandwidth today! 


Headquarters: 
San Jose, CA 
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855.GREP.4.1x% | Contact Us 


992 Compatibility 


online now... 
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